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Abstract 

Rapidly-changing  wartime  environments  and  newly  emerging  global  threats 
necessitate  a  highly  responsive  air  power.  This  responsive  air  power  is  directly  related  to 
the  success  of  Air  Force  logistics  systems  in  generating  sufficient  sorties  required  for 
military  operations.  Briefly,  the  more  efficient  the  logistic  system  is,  the  more  powerful 
the  Air  Force  is. 

Parallel  to  the  developments  in  diagnostic  and  prognostic  technology,  autonomic 
logistics  systems  (AFS)  represent  a  potential  improvement  for  the  aircraft  sortie 
generation  process.  Currently,  Fockheed  Martin  and  the  Joint  Program  Office  are 
developing  a  new  autonomic  logistics  system  for  the  multibillion  F-35  Fightning 
Joint  Strike  Fighter  project,  which  is  named  the  "Autonomic  Fogistics  Information 
System  (AFIS).” 

Generally,  researchers  make  an  analogy  between  the  AFS  and  the  human  body’s 
autonomic  nervous  system  since  both  of  them  monitor,  control,  and  adjust  autonomic 
responses  to  external  stimuli.  Based  on  this  perspective,  AFS  aims  to  switch  the  Air 
Force  logistics  mentality  from  a  reactive  one  into  a  proactive  one  to  achieve  higher  sortie 
generation  rates  and  aircraft  availability. 

The  primary  objective  of  this  thesis  is  to  explore  the  AFS  concept  in  detail  and  to 
investigate  the  F-35’s  sortie  generation  process  through  a  discrete-event  simulation  model 
developed  in  Arena®  Simulation  Software. 
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SIMULATION  OF  AIRCRAFT  SORTIE  GENERATION  UNDER  AN 
AUTONOMIC  LOGISTICS  SYSTEM 


I.  Introduction 


Currently,  Turkey  has  the  second  greatest  armed  forces  in  NATO,  and  the  Turkish 
Air  Force  (TURAF)  is  the  most  deterrent  and  destructive  element  of  this  large  force. 
TURAF  defines  its  vision  as; 

To  become  an  aviation  and  aerospace  power  competing  with  the  age,  which  keeps 
basic  values  of  the  Turkish  Air  Force  alive,  trains  highly-educated  aviator 
manpower,  adopts  a  contemporary  management  approach,  possesses  high 
technology  and  utilizes  it  efficiently,  ensures  deterrence  against  all  kinds  of 
threats  in  its  region,  is  capable  of  conducting  uninterrupted 
separate/joint/combined  operations  anywhere  required  by  national  interests  and 
strengthens  its  superiority  through  national  defense  industry  (Turkish  Air  Force, 
2016). 

TURAF  has  gone  through  many  transitions  since  its  foundation  in  1911  to  fulfill 
its  goal  of  possessing  high  technology  and  utilizing  it  efficiently.  In  particular,  the 
acquisition  of  F-16  Fighting  Falcon  Combat  Aircraft  in  1987  is  assumed  to  be  the  most 
remarkable  of  those  transitions. 

On  July  11,  2002,  when  Turkey  decided  to  join  as  the  seventh  international 
partner  in  the  F-35  Project,  another  important  technology  transition  process  was  initiated 
for  TURAF.  Presently,  Turkey  is  planning  to  procure  100  F-35s,  and  the  first  delivery 
(two  aircraft)  will  be  made  in  2018  (Undersecretariat  of  Defense  Industries,  2016). 

The  F-35’s  unsurpassed  technological  systems  and  unique  stealth  capabilities 
ensure  that  it  will  be  the  future  of  Turkish  national  security  for  decades  to  come.  It  will 
both  introduce  a  new  operational  concept  and  lead  a  logistics  innovation.  Its  cutting  edge 
Autonomic  Logistics  System  will  switch  TURAF’ s  logistics  mentality  from  a  reactive 
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one  into  a  proactive  one,  which  will  enable  higher  sortie  generation  rates  and  aircraft 
availability  in  a  more  cost-efficient  manner. 

This  thesis  develops  a  discrete-event  simulation  model  of  the  F-35’s  sortie 
generation  process  to  provide  decision  makers  with  a  better  understanding  of  the 
Autonomic  Logistics  concept  and  its  impact  on  the  logistics  and  operational  side 
by  enabling  "what-if"  analysis. 

Background 

Rapidly-changing  wartime  environments  and  newly  emerging  global  threats 
necessitate  highly  responsive  air  power  using  cutting-edge  war  technology.  Parallel  to 
this  need,  the  F-35  project  emerged  in  late  1995.  According  to  the  US  Government 
Accountability  Office,  the  F-35  is  the  most  ambitious  and  expensive  weapon  system  in 
DOD’s  history,  with  $400  billion  acquisition  cost  and  $891  billion  sustainment  cost  over 
its  planned  56-year  life  cycle  (Government  Accountability  Office,  2016). 

Responsiveness  of  an  air  power  is  directly  related  to  the  success  of  its  logistics 
system  in  generating  sufficient  sorties  required  for  military  operations.  The  F-35  program 
developers  recognized  the  Autonomic  Logistics  concept  as  the  key  element  to  a  more 
efficient  and  proactive  logistics  system  to  make  the  F-35  a  highly  lethal,  affordable, 
supportable,  and  survivable  aircraft. 

Autonomic  Logistics  (AL)  Concept 

The  term  “Autonomic  Logistics”  (AL)  was  coined  to  describe  an  essentially 
automatic  set  of  processes  to  ensure  maximum  sortie  generation  with  minimum  logistics 
footprint  and  costs,  while  still  maintaining  high  mission  reliability  (Henley,  Currer, 
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Scheuren,  Hess,  &  Goodman,  2000).  It  is  an  inevitable  result  of  the  high-technology 
diagnostic  and  prognostic  applications  in  the  field  of  equipment  support.  In  particular,  the 
successful  implementation  of  an  AL  System  in  the  F-35  fighter,  Autonomic  Logistics 
Information  System  (ALIS),  marked  the  formation  of  AL  mode.  The  AL  system  works 
much  the  same  way  as  a  human  body’s  autonomic  nervous  system.  It  monitors,  controls, 
and  adjusts  autonomic  responses  to  external  stimuli. 

According  to  the  AL  concept,  the  signals  coming  from  special  sensors  embedded 
on  aircraft  are  examined  and  analyzed  constantly  for  the  entire  sortie  generation  process, 
and  fault  detection,  fault  isolation,  fault  prediction  and  reporting  are  made  automatically 
by  the  aircraft  Prognostic  and  Health  Management  (PHM)  system.  The  logistics 
personnel  from  maintenance  to  supply  are  informed  about  the  health  status  of  the  aircraft 
all  the  time.  Therefore,  when  the  aircraft  lands,  the  right  personnel,  equipment,  and 
material  are  at  the  right  place  at  the  right  time,  and  performance  parameters  presumably 
improve  significantly. 

Sortie  Generation  Process 

The  main  purpose  of  the  AL  concept  is  to  improve  the  aircraft  sortie  generation 
process.  Sortie  generation  is  a  cyclic  process  of  flight  related  activities  and  has  been  the 
same  for  many  years  (Faas,  2003).  The  aim  of  this  process  is  to  achieve  a  certain  sortie 
generation  rate  (SGR),  which  is  the  number  of  flight  missions  carried  out  within  a 
specific  timeframe.  SGR  is  a  key  factor  of  a  military  aircraft’s  combat  effectiveness  and 
also  seen  as  an  important  metric  for  senior  commanders  to  evaluate  readiness  of  an  air 
force  to  apply  airpower  (Guoqing,  Hongzhao,  &  Yuanhui,  2010). 
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Literature  reviews  indicate  that  there  is  not  a  unanimous  approach  regarding  the 
starting  point  of  the  sortie  generation  cycle  or  the  steps  within  the  cycle.  However,  it  is 
widely  accepted  that  sortie  generation  is  a  combination  of  inspection,  service,  flight,  and 
maintenance  operations  (Aykiri,  2016;  Faas,  2003;  Guoqing  et  al.,  2010;  Rebulanan, 
2000;  Rossetti  &  McGee,  2006).  In  this  thesis,  sortie  generation  activities  are  gathered 
under  four  groups  as  pre-flight  operations,  sortie,  post-flight  operations,  and  maintenance 
operations.  Pre-flight  operations  are  assumed  as  the  first  stage  of  the  sortie  generation 
process.  Figure  1  represents  the  entire  sortie  generation  process  with  a  different  colored 
background  for  each  main  group. 


Refueling 
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Servicing  , 


/ 


Configuration 


Pre-flight 
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Engine  start, 
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Basic  post-flight 
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Figure  1.  Sortie  Generation  Process  (Partially  adapted  from  (Faas,  2003)) 
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•  Pre-flight  operations:  When  a  flight  mission  is  planned,  the  aircraft  is  refueled 
and  some  servicing  (oil,  tire,  fluid  check  etc.)  is  applied  if  needed.  Then,  mission 
specified  weapons  and  pods  are  installed  onto  the  aircraft.  Since  pre-flight 
operations  are  standardized  actions,  their  completion  times  do  not  vary 
significantly  from  their  average. 

•  Sortie:  Following  the  pre-flight  inspection  the  engine  is  started  and  the  aircraft 
begins  taxiing  onto  the  runway.  Then  it  takes  off,  executes  the  planned  mission, 
and  comes  back  for  landing.  A  sortie’s  duration  depends  on  the  type  of  the 
mission.  For  the  same  mission  type,  usually  it  does  not  vary  from  its  average. 
Expert  views  and  literature  reviews  indicate  that  most  of  the  component  failures 
occur  in  this  phase. 

•  Post-flight  operations:  Following  the  landing,  the  aircraft  is  moved  to  the 
parking  location  and  munitions  are  downloaded.  While  the  aircrew  conducts  their 
debriefing  to  the  maintenance  crew,  concurrently  some  basic  post-flight 
operations  (BPO)  are  applied  to  check  the  aircraft  health  status.  If  there  is  no 
fault  found  during  the  BPO,  the  aircraft  is  routed  to  the  aircraft  pool  for  the  next 
mission.  If  any  fault  is  found,  an  appropriate  maintenance  process  is  initiated  to 
fix  the  problem. 

•  Maintenance  operations:  There  are  three  types  of  maintenance  applied  to  the 
aircraft:  1)  Unscheduled  maintenance;  2)  Scheduled  maintenance;  3)  Depot  level 
maintenance.  Unscheduled  maintenance  is  conducted  to  fix  the  unexpected 
failures  occurring  in  any  step  of  the  sortie  generation  process.  Scheduled 
maintenance  (preventive  maintenance)  is  applied  to  change  the  time-sensitive 
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components  which  reach  a  predetermined  flight  hour-limit,  no  matter  if  there  is  a 
malfunction  or  not.  While  unscheduled  maintenance  may  last  for  several  hours, 
scheduled  maintenance  can  take  a  couple  of  days.  Depot  level  maintenance  is  also 
executed  based  on  the  accumulated  flight  hours.  Some  modifications  programs 
are  applied  to  bring  the  currently  fielded  aircraft  to  their  expected  airframe 
structural  lifespan  and  usually  require  several  months  (The  Office  of  The 
Director,  2015).  Figure  2  briefly  represents  the  maintenance  process  flow. 


Figure  2.  The  Flow  of  Unscheduled  and  Scheduled  Maintenance  (Adapted  from  Faas,  2003) 


Literature  reviews  indicates  that  an  ALS  is  expected  to  make  the  most  significant 
contribution  to  the  maintenance  operations  by  introducing  a  proactive  approach  over  the 
reactive  approach  used  today.  Therefore,  the  main  focus  of  our  simulation  model  is  the 
maintenance  processes  portion  of  the  sortie  generation  cycle,  with  particular  emphasis  on 
unscheduled  maintenance.  Impacts  of  ALS  on  maintenance  operations  are  explained  in 
detail  in  the  following  chapters. 
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Previous  Work 


Due  to  its  great  value  to  a  military  force  in  terms  of  cost,  sortie  generation,  and 
aircraft  availability,  the  Autonomic  Logistics  System  (ALS)  has  become  a  popular 
subject  of  scientific  research.  An  increasing  number  of  simulations  and  analytical  studies 
are  being  conducted  to  analyze  the  AL  in  depth.  Literature  reviews  showed  that  some 
pioneering  simulation  studies  regarding  the  ALS  and  sortie  generation  process  were 
conducted  by  former  AFIT  students. 

While  Rebulanan  (2000)  and  Malley  (2001)  used  an  object-oriented  design  with 
JAVA®  and  Silk®  programming  languages,  Yager  (2003)  built  a  closed  queuing  model 
and  Faas  (2003)  developed  a  discrete  event  simulation  model  with  Arena®  software  to 
investigate  the  possible  impacts  of  the  ALS  on  the  sortie  generation  process. 

These  studies  were  conducted  in  the  early  2000s,  a  time  when  the  F-35  project 
and  ALS  were  in  their  infancy,  and  mainly  investigated  ALS  from  a  conceptual 
perspective.  Researchers  were  unable  to  use  real  F-35  data  due  to  the  fact  that  a  working 
system  didn’t  exist,  and  none  of  them  included  Learning  Curves  (LCs)  and  reliability 
growth  concepts.  Therefore,  this  research  attempts  to  fill  that  gap  in  the  scientific  field  by 
building  a  simulation  model  based  as  much  as  possible  on  actual  F-35  data.  Moreover, 
LCs  and  reliability  growth  are  included  into  the  model  logic  to  obtain  more  realistic 
outputs. 

Problem  Statement 

Although  F-35  aircraft  will  be  the  main  TURAF  combat  element  in  the  near 
future,  its  logistics  properties  and  capabilities  are  not  well  known.  Decision  makers  and 
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planners  within  the  TURAF  need  to  recognize  and  appreciate  the  potential  impacts  of  the 
AL  concept  to  manage  F-35  fleet  more  effectively  and  efficiently. 

Purpose  Statement 

In  this  study,  the  ALS  is  investigated  in  detail  and  a  discrete-event  simulation 
model  for  the  F-35  sortie  generation  process  is  developed  using  Arena®  Simulation 
Software.  After  collecting  the  data  from  the  simulation  model,  statistical  analyses  are 
performed  through  JMP®  Software  in  designed  experiments  to  determine  the  ALS’s 
impacts  on  the  measures  of  performance  (MOPs)  of  the  F-35’s  sortie  generation  process. 

Research  Objectives  and  Questions 

The  Sortie  generation  process  includes  many  parameters,  which  makes  it 
extremely  difficult  to  exhaustively  analyze.  Therefore,  the  main  objective  of  this  research 
is  to  provide  a  useful  simulation  model  for  decision  makers  to  recognize  the  key  factors 
within  the  ALS  and  their  potential  impacts  on  MOPs  of  the  F-35’s  sortie  generation 
process.  The  following  questions  capture  the  main  focus  of  our  research: 

1.  What  impact  does  the  maturity  level  of  the  PHM  system  have  on  the  MOPs  of  the 
F-35’s  sortie  generation  process? 

2.  What  impact  do  learning  curves  have  on  the  MOPs  of  the  F-35’s  sortie  generation 
process? 

3.  What  impact  does  reliability  growth  have  on  the  MOPs  of  the  F-35’s  sortie 
generation  process? 
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Scope  and  Assumptions 

Presently,  there  are  three  variants  of  the  F-35.  This  research  specifically 
investigates  the  sortie  generation  process  of  the  conventional  F-35  variant,  the  F-35A, 
since  it  is  the  only  variant  that  will  be  procured  by  Turkey. 

Additionally,  the  F-35  aircraft  is  a  very  complex  system  consisting  of  many 
subsystems,  and  its  availability  is  directly  dependent  on  the  functionality  of  all  those 
subsystems.  However,  modelling  a  sortie  generation  process  with  all  those  subsystems 
would  be  too  complicated  and  time  consuming,  since  each  of  them  might  have  different 
failure  rates  and  repair  times.  Therefore,  for  simplicity,  the  scope  of  this  research  is 
limited  to  only  mission  critical  failures  encountered  by  the  F-35A  variant.  No  LRC  (Line 
Replaceable  Component)  specification  is  made  and  all  LRCs  are  assumed  to  have  similar 
failure  rates  and  repair  times.  Moreover,  only  diagnostic  capability  of  the  F-35  is 
modeled,  since  prognostic  capability  is  not  yet  functional. 

Summary 

This  chapter  described  the  rationale  behind  the  research  with  regards  to  the  F-35 
project,  Autonomic  Logistics,  and  aircraft  sortie  generation,  and  provided  an  overview  of 
the  problem  statement,  purpose  statement,  research  objective,  research  questions,  scope, 
and  assumptions.  Chapter  II  presents  reviews  of  the  existing  literature  on  the  sortie 
generation  process.  Chapter  III  describes  the  data  used  to  meet  the  research  objectives,  as 
well  as  the  data  analysis  and  model  development.  Chapter  IV  provides  results  of  the 
research,  while  Chapter  V  provides  conclusions  and  recommendations  for  further 
research. 
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II.  Literature  Review 


Introduction 

This  chapter  examines  prior  researches  conducted  in  the  area  of  the  sortie 
generation  process,  Autonomic  Logistics,  and  Learning  Curves.  Also,  the  F-35  project  is 
discussed  from  a  logistics  basis.  Furthermore,  the  present  situation  of  the  F-35’s 
Autonomic  Logistics  Information  System  is  put  forward  in  the  view  of  official  reports 
presented  by  the  United  States  Government  Accountability  Office  (GAO). 

Sortie  Generation  Process 

As  described  in  Chapter  1,  sortie  generation  is  a  combination  of  many  activities 
dedicated  to  produce  desired  number  of  sorties  within  a  limited  time  period.  Due  to  its 
paramount  importance  for  an  air  force’s  combat  effectiveness  and  deterrence,  sortie 
generation  has  been  a  popular  subject  of  scientific  research  for  many  years.  Some  key 
studies  conducted  in  this  field  are  presented  below. 

Guoqing  et  al.  (2010)  developed  an  approximate  analytical  method  producing 
highly  similar  results  to  those  obtained  from  long  simulations.  Their  study  showed  that 
sortie  generation  rate  was  the  key  parameter  for  a  military  aircraft’s  combat  effectiveness. 

MacKenzie  et  al.  (2012)  also  examined  the  relationship  between  the  number  of 
sorties  flown  and  Combat  Mission  Readiness  (CMR).  They  assumed  CMR  was  a  key 
metric  for  senior  commanders  to  evaluate  readiness  of  an  air  force  to  apply  air  power. 
They  demonstrated  that  different  mixes  of  maintenance  personnel  skill  levels 
significantly  affect  the  sortie  generation  rate. 
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Iwata  and  Mavris  (2013)  focused  on  supply  activities  within  the  sortie  generation 
process.  They  put  forward  the  relationship  between  mission  capable  (MC)  rate  and 
average  part  delivery  time  by  conducting  a  case  study  using  a  Python  simulation  tool. 
They  showed  that  MC  rate  began  to  decrease  after  the  delivery  time  grew  beyond  1.0 
day,  and  flattened  off  at  a  MC  rate  of  0.7  from  1.4  to  1.9  days.  After  that  point,  MC  rate 
fell  rapidly. 

Harris  (2002)  noted  that  the  Logistics  Composite  Model  (LCOM),  currently  being 
used  by  USAF  to  calculate  sortie  rates,  requires  a  great  amount  of  input  data  and 
processing  time.  Therefore,  LCOM  hinders  the  commander’s  flexibility,  responsiveness, 
and  ability  to  create  alternative  options.  Thus,  he  proposed  the  Sortie  Generation  Rate 
(SGR)  model  which  was  a  generic  sortie  model  with  simple  operational  input  and  quick 
turnaround.  The  SGR  model  generated  sortie  rates  that  were  close  to  the  actual  sortie 
rates. 

Lastly,  Manuel  D.  Rossetti  and  Joshua  B.  McGee  (2006)  demonstrated  that 
simulation  could  provide  valuable  information  for  decision-making  at  the  unit  level  and 
provide  much  needed  assistance  in  the  generation  and  execution  of  a  weekly  flying 
schedule.  Thus,  their  model  allowed  the  unit  level  logistics  planners  to  compare 
alternative  schedules  and  perform  what-if  analysis. 

Literature  reviews  show  that  the  impacts  of  the  Autonomic  Logistics  (AL)  on 
sortie  generation  process  became  another  focus  topic  for  researchers  when  the  F-35 
project  was  introduced  in  mid-1990.  Some  pioneering  simulation  studies  in  this  area  were 
conducted  by  AFIT  graduates. 
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First,  Rebulanan  (2000)  created  a  computer  simulation  model  of  the  ALS,  called 
ALSim,  using  object-oriented  design  with  Java  and  Silk  programming  languages  to 
compare  the  ALS  with  the  existing  logistics  system.  His  study  indicated  that  the  ALS 
could  improve  the  system  performance  in  terms  of  aircraft  availability,  sortie  generation 
rate,  and  time  waiting  for  parts.  Following  Rebulanan’ s  study,  Malley  (2001)  added  the 
reality  of  false  alarms  into  Rebulanan’ s  ALSim  to  generate  more  realistic  failure 
detection  times  in  order  to  make  the  simulation  more  useful  for  decision  making. 

Next,  Faas  (2003)  built  a  discrete  event  simulation  model  of  sortie  generation 
with  Arena®  software  to  investigate  the  ALS  concept.  His  model  also  indicated  that  the 
ALS  equipped  aircraft  could  perform  more  effectively  than  a  non-ALS  aircraft  up  to  a 
point.  In  the  best  case  scenario,  in  which  false  alarms  were  at  a  minimum  level,  there  was 
an  8%  improvement  in  Mission  Capable  rate.  At  the  other  factor-level  combinations,  the 
ALS  performed  marginally  better  or  even  worse  than  the  non-ALS  aircraft. 

Cassady  et  al.  (2006)  developed  a  simulation  model  with  Arena  to  explore  the 
impacts  of  diagnostic  and  prognostic  errors  on  fleet  performance  and  compare  prognostic 
to  scheduled  maintenance.  Focusing  on  type-1  and  type-2  errors,  they  showed  that 
prognostics  could  be  an  effective  tool  in  some  cases  (even  in  the  presence  of  significant 
prognostic  errors)  and  a  very  ineffective  tool  in  other  cases.  In  some  cases,  prognostic 
errors  could  make  a  situation  worse  than  a  "run  to  failure”  policy  (unscheduled 
maintenance)  would  do. 

To  sum  up,  the  literature  reviews  demonstrated  that  simulation  is  a  useful  tool  to 
investigate  aircraft  sortie  generation  processes.  Moreover,  sortie  generation  rate  and 
aircraft  availability  are  the  two  most  commonly  used  sortie  generation  performance 
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parameters.  However,  none  of  the  studies  used  real-world  F-35  data.  They  mostly 
investigated  sortie  generation  from  a  theoretical  basis.  Therefore,  developing  a  simulation 
model  based  on  real-world  F-35  data  may  provide  significant  contributions  to  this 
research  area. 

The  F-35  Project 

The  second  topic  investigated  through  the  literature  reviews  was  the  F-35  project. 
According  to  the  GAO  reports,  the  F-35  is  the  most  ambitious  and  expensive  weapon 
system  in  DOD’s  history  with  sustainment  costs  comprising  the  vast  majority  of  DOD’s 
$1.3  trillion  cost  estimate.  It  is  a  joint,  multinational  acquisition  intended  to  develop  and 
field  a  family  of  next-generation  strike  fighter  aircraft  for  the  United  States  Air  Force, 
Navy,  and  Marine  Corps,  and  eight  international  partners  (United  Kingdom,  Italy,  the 
Netherlands,  Turkey,  Canada,  Australia,  Denmark,  and  Norway).  Lockheed  Martin  is  the 
primary  aircraft  contractor  and  Pratt  &  Whitney  is  the  engine  contractor  (Government 
Accountability  Office,  2016). 

The  F-35  project  is  currently  in  the  low-rate  production  stage  with  full-rate 
production  planned  to  start  by  2019.  The  timeline  of  major  events  in  the  F-35  program  is 
presented  in  Table  1. 
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Table  1.  Timeline  of  Major  Events  in  the  F-35  Program  (Government  Accountability  Office,  2014) 


2001 :  Milestone  6  is  approved  and 
Lockheed  Martin  is  selected  as  the 
prime  contractor." 


2012:  Program 
establishes  new 
acquisition 
baseline  and 
recertifies 
Milestone  B  after 
a  cost-growth 
breach. 


2016:  Air 
Force  to 
declare  IOC. 


2019:  Planned 
milestone  decision 
to  enter  full-rate 
production. 


The  program  begins  development. 


2015:  Program  to  2018:  Navy  to  declare  IOC 


begin  operational 
testing,  and  Manne 
Corps  to  declare 
Initial  Operalional 
Capability  (100.“ 


Source  GAO  analyse  ol  Department  ot  Detense  information  |  GAO  14  778 


According  to  the  DOD,  there  will  be  three  variants  of  the  F-35: 

•  The  conventional  takeoff  and  landing  (CTOL)  variant,  designated  the  F-35A, 
is  a  multirole,  stealthy  strike  aircraft  replacement  for  the  Air  Force’s  F-16  Falcon 
and  the  A- 10  Thunderbolt  II  aircraft,  and  complements  the  F-22A  Raptor.  Turkey 
will  procure  the  CTOL  variant. 

•  The  short  takeoff  and  vertical  landing  (STOVL)  variant,  designated  the  F- 
35B.  is  a  multirole,  stealthy  strike  fighter  that  replaces  the  Marine  Corps’  F/A- 
18C/D  Hornet  and  AV-8B  Harrier  aircraft. 

•  The  carrier-suitable  variant  (CV),  designated  the  F-35C  provides  the  Navy  a 
multirole,  stealthy  strike  aircraft  to  complement  the  F/A-18. 

McCollom  and  Worth  (2011)  state  that  the  F-35  aircraft  vision  rests  on  four  main 
pillars:  Affordability,  Lethality,  Supportability,  and  Survivability  (see  Figure  3).  The 
program  has  a  unique  commitment  to  the  creation  of  a  new  form  of  aircraft  and 
operational  systems,  with  a  fundamental  and  essential  focus  on  two  of  the  four  program 
“pillars”-  Supportability  and  Affordability. 
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Figure  3.  F-35  Aircraft  Vision  (McCollom,  2011) 


The  supportability  and  affordability  of  the  F-35  are  directly  related  to  the 
sustainment  costs,  since  they  are  the  most  significant  cost  driver  of  the  F-35  program. 
The  sustainment  costs  consist  of  Operation  and  Support  (O&S)  costs  incurred  from  the 
initial  system  deployment  through  the  end  of  system  operations,  and  include  all  costs  of 
operating,  maintaining,  and  supporting  the  fielded  system.  The  F-35  program  office 
develops  an  annual  estimate  for  the  O&S  costs  of  maintaining  and  supporting  the  F-35 
for  56  years.  In  its  most  recent  estimate  (2014),  the  program  office  estimates  cost  at  about 
$891  billion  to  sustain  the  entire  F-35  fleet  over  its  life  cycle  (Government 
Accountability  Office,  2016). 

These  financial  estimates  related  to  the  F-35’s  sustainment  highlight  the 
importance  of  a  new  logistics  system  that  should  be  far  more  efficient  than  the  legacy 
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logistics  systems.  The  F-35  program  developers  recognize  the  Autonomic  Logistics 
concept  as  the  key  element  to  a  more  efficient  logistics  system  which  will  make  the  F-35 
a  highly  lethal  and  affordable  aircraft. 

Autonomic  Logistics  (AL)  Concept 

The  AL  concept  is  the  inevitable  result  of  high-tech  diagnostic  and  prognostic 
advances  in  the  field  of  equipment  support.  It  aims  to  achieve  condition-based 
maintenance  by  using  the  health  status  data  coming  from  special  sensors  embedded  onto 
the  aircraft. 

The  AL  concept  has  four  major  parts:  1)  Prognostics  and  Health  Management 
(PHM);  2)  Joint  Distributed  Information  System  (JDIS);  3)  Technology-enabled 
maintainer;  4)  Responsive  logistics  infrastructure  (Henley  et  al.,  2000). 

PHM  is  a  kind  of  on-board  artificial  autonomic  nervous  system  which  is  vital  for 
AL  operations.  Through  the  use  of  intelligent  reasoners,  PHM  detects,  isolates,  and 
predicts  failures  or  triggers  a  single  maintenance  action  in  the  event  of  unpredicted  failure 
(Henley  et  al.,  2000).  Henley  et  al.  defined  key  benefits  of  PHM  as: 

•  Improved  safety, 

•  Improved  sortie  generation, 

•  Triggering  of  AL  functions, 

•  Reduced  life  cycle  costs, 

•  Reduced  logistics  footprint, 

•  Triggering  of  system  reconfiguration  to  achieve  mission  reliability, 
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•  Providing  advanced  onboard  diagnostics  and  testability  to  reduce  the  skillset 

needed  by  maintainers. 

JDIS  is  an  advanced  information  technology  providing  decision  support  tools  and 
an  effective  communication  network  linking  aircraft  with  the  logistics  infrastructure  to 
provide  proactive  support  and  enable  remote  maintenance  when  needed.  The  information 
fusion  capability  of  the  PHM  system  allows  JDIS  to  output  and  pass  on  actions  and 
recommendations  rather  than  just  data  (Hess,  Calvello,  &  Dabney,  2004).  By  fusing  the 
information  coming  from  sensors,  it  produces  the  following  outputs: 

•  Maintenance  Information/knowledge, 

•  Supply  chain  management  information, 

•  Health  and  usage  information, 

•  Forecast  aircraft  availability  data, 

•  Best  use  of  resource  recommendations, 

•  Training  Management  (Henley  et  al.,  2000). 

With  the  help  of  the  outputs  above,  the  following  tasks  are  automatically 
performed  through  JDIS:  Mission  planning,  maintenance  action  scheduling,  ordering  of 
spare  parts,  scheduling  of  flight  and  maintenance  training,  assignment  of  specific  pilots  to 
specific  missions  based  upon  experience  and  readiness,  assigning  specific  aircraft  to 
specific  missions  based  upon  aircraft  availability  and  capability,  and  storing  maintenance, 
training,  spare  part,  and  logistic  information  in  the  data  warehouse  (Hess  et  al.,  2004). 

The  maintainer  in  the  AL  concept  is  enabled  with  a  full  set  of  technological  tools 
to  prepare  the  aircraft  for  its  next  sortie  in  the  most  effective  way.  The  maintainer’ s 
toolset  consists  of: 
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•  Comprehensive  knowledge  of  the  actual  aircraft  health  before  beginning  work, 

•  Appropriate  and  timely  training  to  conduct  the  task, 

•  All  the  necessary  material  on  hand  before  commencement  of  work, 

•  Interactive  guidance  available  in  real  time  to  provide  supplementary  information 

as  required  (Henley  et  al.,  2000). 

A  technologically  enabled  maintainer  is  capable  of  efficiently  and  effectively 
maintaining  the  F-35  with  less  specialized  training  and  more  “on  the  spot”  training.  This 
allows  the  use  of  fewer  maintainers,  cross  trained  over  many  sub-systems  (Hess  et  al., 
2004). 

Finally,  a  flexible  and  responsive  logistics  infrastructure  is  needed  to  get  full 
benefit  from  the  substantial  PHM  capability,  technologically  enabled  maintainer,  and 
highly  capable  JDIS. 

Autonomic  Logistics  Information  System  (ALIS) 

The  F-35  program  developers  recognize  the  AL  concept  as  the  key  enabler  to  a 
highly  supportable  and  affordable  fighter.  The  AL  implementation  in  the  F-35  fighter 
aircraft  is  named  the  “Autonomic  Logistics  Information  System”  (ALIS),  which  was 
also  called  the  Joint  Distributed  Information  System  (JDIS)  by  many  previous 
researchers.  ALIS  is  one  of  three  major  components  that  make  up  the  F-35  air  system, 
along  with  the  aircraft  and  the  engine,  and  comprises  both  hardware  and  software 
(Government  Accountability  Office,  2014). 

Lockheed  Martin,  the  prime  contractor  of  the  F-35  project,  describes  ALIS  in  its 
official  website  as:  “ALIS  serves  as  the  information  infrastructure  for  the  F-35, 
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transmitting  aircraft  health  and  maintenance  action  information  to  the  appropriate  users 
on  a  globally-distributed  network  to  technicians  worldwide.  ALIS  receives  Health 
Reporting  Codes  via  a  radio  frequency  downlink  while  the  F-35  is  still  in  flight;  this 
enables  the  pre-positioning  of  parts  and  qualified  maintainers  so  that  when  the  aircraft 
lands,  downtime  is  minimized  and  efficiency  is  increased.”  (Lockheed  Martin,  2016). 

ALIS  has  three  main  hardware  components: 

•  The  Autonomic  Logistics  Operating  Unit  (ALOU):  The  ALOU  is  the  computer 
server  that  all  F-35  data  ultimately  are  sent  through  and  it  supports 
communications  with  and  between  the  government  and  the  contractor’s  systems. 

•  The  Central  Point  of  Entry  (CPE):  The  CPE  is  configured  to  provide  software 
and  data  distribution  for  the  entire  F-35  fleet  in  the  United  States,  enables 
interoperability  with  national  (government)  systems  at  the  country  level,  and 
enables  ALIS  data  connectivity  between  bases.  Each  international  partner 
operating  F-35  aircraft  is  expected  to  have  its  own  CPE  at  other  locations. 

•  The  Standard  Operating  Unit  (SOU):  SOUs  provide  all  ALIS  capabilities  to 
support  flying,  maintenance,  and  training.  They  also  provide  access  to 
applications  to  operate  and  sustain  the  aircraft  (Government  Accountability 
Office,  2016). 

General  architecture  and  primary  applications  of  the  ALIS  are  presented  in  Table 
2.  According  to  F-35  Program  Office’s  assessment  of  functionality  status  as  of  January 
2016,  green  applications  (JTD,  LOHAS)  have  no  issues,  yellow  applications  (AVD/H, 
CMMS,  MVI,  Propulsion,  SCM)  have  minor  issues,  and  red  applications  (OMS,  TMS) 
have  major  issues. 
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Table  2.  Primary  ALIS  Applications  and  the  F-35  Program  Office’s  Assessment  of  Their 
Functionality  Status  as  of  January  (Government  Accountability  Office,  2016) 


Autonomic  Logistics  Information  Systems  (ALIS)  is  the 

primary  sustainment  tool  for  the  F-35  and  is  intended  to 
predict  maintenance  and  supply  issues,  automate  logistics 
support  processes,  and  provide  decision  aids  to  help  reduce 
life-cycle  sustainment  costs  and  improve  force  readiness. 


Source:  GAO  presentation  of  Department  of  Defense  analysts.  |  GAO-16-439 


DOD  has  estimated  ALIS  related  total  costs  to  be  about  $16.7  billion  over  the  F- 
35’s  56-year  life  cycle.  However,  a  2013  DOD  commissioned  plan  found  that  schedule 
slippage  and  functionality  problems  with  ALIS  could  lead  to  $20-100  billion  in  additional 
costs. 
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Table  3.  Autonomic  Logistics  and  Information  System  Costs  (Government  Accountability  Office, 

2016) 


Cost  element 

Then-year  dollars  in 
millions 

Development  costs 

Expended  as  of  December  2015 

505 

Estimated  development  costs  remaining  through  2017 

57 

Subtotal 

$562 

Estimated  procurement  costs 

Hardware 

931 

Spare  parts 

147 

Subtotal 

$1,078 

Estimated  sustainment  costs 

Contractor  support3 

8,050 

Technology  refresh13 

3,850 

Hardware  maintenance  agreements 

1,603 

Software  licensing  agreements 

1,598 

Subtotal 

$15,101 

Total 

$16,741 

Source:  GAO  analysis  of  F-35  program  office  data.  |  GAO-16-439 


Maintenance  Concepts  and  ALIS 

Faas  (2003)  defines  maintenance  as  the  heart  of  flight  line  operations  due  to  its 
paramount  importance  in  generating  a  desired  number  of  sorties.  It  is  one  of  the  most 
significant  cost  drivers  in  aircraft  sustainment.  According  to  the  DOD,  maintenance  costs 
hold  almost  30  percent  of  the  F-35’s  sustainment  costs  (Government  Accountability 
Office,  2014). 

Aircraft  maintenance  has  evolved  over  time  from  a  “fix  it  when  it  breaks”  policy 
to  a  condition-based  maintenance  concept  (Vandawaker,  2015).  Table  4  presents  a 
categorical  breakdown  of  maintenance  approaches  and  their  attributes. 
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Table  4.  Maintenance  Concepts  (Vandawaker,  2015) 


Maintenance  Approaches 


Category 

Reactive 

Proactive 

Sub- 

Category 

Run-to-fail 

Preventive 

Predictive 

Fix  when  it 
breaks 

Scheduled  maintenance 

Condition-based 

maintenance- 

diagnostic 

Condition-based 

maintenance- 

prognostic 

When 

Scheduled 

No 

scheduled 

maintenance 

Maintenance  based  on  a 
fixed  time  schedule  for 
inspect,  repair  and  overhaul 

Maintenance 
based  on  current 

condition 

Maintenance  based  on 
forecast  of  remaining 
equipment  life 

Why 

Scheduled 

N/A 

Intolerable  failure  effect 
and  it  is  possible  to  prevent 
the  failure  effect  through  a 
scheduled  overhaul  or 
replacement 

Maintenance 
scheduled  based 
on  evidence  of 
need 

Maintenance  need  is 
projected  as  probable 
within  mission  time 

How 

Scheduled 

N/A 

Based  on  the  useful  life  of 
the  component  forecasted 
during  design  and  updated 
through  experience 

Continuous 
collection  of 
condition 
monitoring  data 

Forecasting  of 
remaining  equipment 
life  based  on  actual 
stress  loading 

Kind  of 

Prediction 

None 

None 

On-  and  off- 
system.  near- 
real-time  trend 
analysis 

On-  and  off-system, 
real-time  trend 
analysis 

Maintenance  Process  within  Legacy  Logistics  System 

In  the  current  situation,  legacy  aircraft  are  supported  by  two  types  of 
maintenance:  unscheduled  maintenance  and  scheduled  (preventive)  maintenance.  The 
ground  crew  is  only  notified  of  a  fault  either  prior  to  landing  if  the  pilot  radios  in  or  on 
the  ground  after  engine  shutdown  (Faas,  2003).  Fault  diagnosis  is  mainly  carried  out 
during  post  flight  servicing,  inspection  and  aircrew  briefing.  Once  the  problem  is 
diagnosed,  required  part/parts  are  ordered  from  supply.  After  the  parts  are  received, 
unscheduled  maintenance  is  conducted  to  fix  the  problem.  Also,  maintenance  personnel 
must  document  the  entire  process. 


22 


Regarding  preventive  maintenance,  Time  Change  Item  (TCI)  replacements  are 
conducted  based  on  accumulated  flight  hours  of  the  critical  parts,  not  based  on  part 
condition  (Faas,  2003).  Once  a  part  reaches  its  limit  of  accumulated  flight  hours,  it  is 
replaced  or  repaired  no  matter  if  it  is  still  functional  or  not. 

Maintenance  Process  within  ALIS 

As  stated  in  the  "Autonomic  Logistics  Concept"  section,  theoretically  many 
significant  changes  take  place  within  the  F-35  sortie  generation  process  with  a  full- 
functional  Autonomic  Logistics  and  Information  System  (ALIS).  Supposedly,  ALIS 
provides  the  most  considerable  improvement  on  the  maintenance  step  by  substituting  a 
proactive  approach  for  the  existing  reactive  approach. 

First  of  all,  health  status  of  the  F-35  aircraft  is  monitored  by  ALIS  for  the  entire 
sortie  generation  process.  The  signals  coming  from  special  sensors  on  the  aircraft  are 
fused  via  some  reasoners  and  PHM  data  (including  time,  status  of  subsystems  and  other 
health  related  information)  are  transferred  to  the  maintenance  and  supply  units 
concurrently.  Fault  detection,  fault  isolation  and  documentation  are  done  automatically. 
Therefore,  maintainers  are  enabled  to  diagnose  the  failures  more  easily  and  quickly,  and 
the  amount  of  diagnostic  equipment  and  time  are  reduced  considerably.  Since  the  right 
personnel,  the  right  equipment,  and  the  right  part  are  ready  at  the  right  place  at  the  right 
time,  costs  and  delays  of  maintenance  are  minimized  significantly. 

Second,  prognostic  capabilities  of  ALIS  replace  the  existing  preventive 
maintenance  with  condition-based  maintenance.  Therefore,  the  time-change  items  which 
are  normally  replaced  according  to  predetermined  flight  hours  in  the  preventive 
maintenance  concept  will  be  only  replaced  when  they  become  non-functional. 
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Additionally,  ALIS  provide  maintainers  with  timely  knowledge  of  impending 
failures.  Therefore,  opportunistic  maintenance  is  possible  by  grouping  multiple 
maintenance  actions  at  a  single  time,  while  the  aircraft  is  already  down.  For  instance,  a 
hypothetical  aircraft  is  down  for  a  routine  engine  wash.  While  it  is  being  attended  to,  the 
prognostics  system  informs  maintainers  that  the  primary  auxiliary  power  unit  has  begun 
to  degrade  and  needs  to  be  replaced  within  the  next  15  flight  hours.  It  also  informs  the 
maintainers  that  the  oil  in  the  engine  is  beginning  to  show  signs  of  coking  and  has  an 
undesirably  high  content  of  fragments.  Hence,  all  three  maintenance  actions  can  be  taken 
care  of  with  a  single  downing  of  the  aircraft,  vice  three  separate  maintenance  actions 
which  would  keep  the  plane  out  of  commission  for  some  time  (Hess  &  Fila,  2002). 

However,  in  practice,  ALIS  is  far  from  providing  all  of  the  theoretical  benefits 
mentioned  above.  According  to  some  official  reports  prepared  by  Government 
Accountability  Office  and  The  Office  of  The  Director,  Operational  Test  and  Evaluation 
(DOT&E),  ALIS  has  experienced  recurring  problems,  including  user  issues  and  schedule 
delays.  The  integration  of  ALIS  capabilities — which  are  fielded  in  increments — has  been 
repeatedly  delayed.  Additionally,  ALIS’s  prognostic  capability  still  is  not  functional  and 
its  diagnostic  system  has  not  reached  full  functionality  yet  (Government  Accountability 
Office,  2014;  The  Office  of  The  Director,  2015). 

Action  Request  (AR)  System  within  ALIS 

Currently,  maintenance  personnel  track  issues  with  ALIS  through  an  internal 
reporting  mechanism  called  the  Action  Request  (AR)  System,  which  allows  users  in  the 
field  to  identify  problems  with  the  system  for  potential  fixes  (Government  Accountability 
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Office,  2014).  Upon  landing,  a  computer  system  is  attached  to  the  aircraft  and  gathers  all 


the  information  needed  to  decide  whether  the  aircraft  has  a  maintenance  issue  or  if  it  is 


ready  to  fly  again.  If  there  is  a  problem  and  a  known  fix  is  not  available  in  the  F-35’s 
Joint  Technical  Data  (JTD),  an  Action  Request  (AR)  is  initiated  by  the  maintenance 
personnel  and  sent  to  the  Lockheed  Martin  engineers  for  tailored  instructions  to  fix  the 
discrepancy.  After  an  appropriate  resolution  is  reached,  maintenance  personnel  fix  the 
problem  (Colbacchini,  Gahafer,  Mcevoy,  &  Park,  2016).  Figure  4  represents  the  overall 


issue-resolution  process. 


Diagnostic  health  codes  are  downloaded 
from  the  plane  to  Autonomic  Logistics 
Information  System  (ALIS). 


Solution  sent 
back  to 

maintainer  and 
ALIS  is  updated 


Maintainer 
retrieves  codes 
from  ALIS  and 
reports  any  issues 
to  the  contractor. 


Contractor 
receives  issue 
reports  from 
operations. 


Sample  of  ALIS  capabilities 


-Assess  usage 

■  Perform  aircraft  maintenance 

•  Perform  air  vehicle  post-  and 
preflight  servicing 

•  Plan  mission 

•  Process  diagnostics  data 

•  Process  sustainment  data 

•  Provide  logistics  data 


Source:  GAO  analysis  of  Department  of  Defense  information;  Lockheed  Martin.  |  GAO-14-778 


Figure  4.  ALIS  Issue-Resolution  Process  (Government  Accountability  Office,  2014) 
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The  AR  System  aims  to  route  and  monitor  ARs  through  the  maintenance  cycle 
efficiently.  However,  its  current  performance  does  not  meet  the  desired  level.  Using 
Microsoft  Excel’s  YASAI  add-in,  Colbacchini  et  al.  (2016)  developed  a  discrete  event 
simulation  model  to  determine  how  to  minimize  the  time  in  the  AR  system.  They  found 
that  an  AR  for  the  most  severe  problems  took  an  average  of  17  days  to  navigate  through 
the  AR  system,  which  did  not  meet  the  Air  Force  standards.  They  concluded  that 
maintenance  personnel  should  increase  training  on  the  AR  System  and  Lockheed  Martin 
should  hire  more  engineers  to  reduce  the  process  time  of  ARs.  A  2015  DOT&E  report 
and  a  2016  GAO  report  supported  the  findings  of  Colbacchini  et  al.  by  indicating  that 
ALIS’s  AR  process  is  insufficient  and  problematic  (Government  Accountability  Office, 
2016;  The  Office  of  The  Director,  2015). 

The  literature  reviews  showed  that  the  AR  system  and  its  effects  on  the  sortie 
generation  process  have  not  been  thoroughly  investigated.  Since  it  holds  an  important 
place  within  the  F-35’s  current  maintenance  activities  and  causes  considerable  delays,  it 
is  included  in  our  simulation  model  in  order  to  obtain  more  realistic  results. 

Prognostic  Health  Management  (PHM)  within  ALIS 

ALIS’s  PHM  system  has  three  major  components:  fault  and  failure  management 
(diagnostic  capability),  life  and  usage  management  (prognostic  capability),  and  data 
management. 

According  to  a  2015  DOT&E  report,  the  F-35’s  PHM  diagnostic  and  data 
management  capabilities  remain  immature  and  the  program  does  not  yet  plan  to  integrate 
prognostic  capabilities.  Diagnostic  capabilities  demonstrate  poor  accuracy,  low  detection 
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rates,  and  a  high  false  alarm  rate.  Table  5  compares  specific  diagnostic  measures  from  the 
Operational  Requirements  Document  (ORD)  with  current  values  of  performance  through 
June  2015  (The  Office  of  The  Director,  2015). 


Table  5.  Metrics  of  F-35's  Diagnostic  Capabilities 


METRICS  OF  DIAGNOSTIC  CAPABILITY 

(6-month  rolling  window  as  of  June  201 5.  Data  provided  by  the  Program  Office  are  considered  "preliminary"as  they  have  not  completed 

the  formal  adjudication  process  by  the  data  review  board.) 

Diagnostic  Measure 

Threshold 

Requirement 

Demonstrated  Performance 

Block  1 

Block  2 

Block  3 

Developmental  Test  and  Production  Aircraft 

Fault  Detection  Coverage  (percent  mission  critical  failures  detectable  by  PHM) 

N/A 

65 

73 

84 

Fault  Detection  Rate  (percent  correct  detections  for  detectable  failures) 

98 

65 

73 

85 

Fault  Isolation  Rate  (percentage):  Electronic  Fault  to  One  Line  Replaceable 
Component  (LRC) 

90 

68 

69 

72 

Fault  Isolation  Rate  (percentage):  Non-Electronic  Fault  to  One  LRC 

70 

76 

72 

79 

Fault  Isolate  Rate  (percentage):  Non-Electronic  Fault  to  3  or  Fewer  LRC 

90 

82 

87 

87 

Production  Aircraft  Only 

Mean  Flight  Hours  Between  False  Alarms 

50 

0.20 

0.60 

0.18 

Mean  Flight  Hours  Between  Flight  Safety  Critical  False  Alarms 

450 

1,360 

543 

170 

Accumulated  Flight  Hours  for  Measures 

N/A 

1,360 

4,886 

1,360 

Ratio  of  False  Alarms  to  Valid  Maintenance  Events 

N/A 

44:1 

16:1 

1079:1 

Poor  diagnostic  performance  increases  maintenance  downtime.  Maintainers  often 
conduct  built-in  tests  to  see  if  the  fault  codes  detected  by  the  diagnostics  are  true  faults. 
False  alarms  lead  to  unnecessary  maintenance  actions.  These  actions  increase 
maintenance  man-hours  per  flight  hour,  which  in  turn  can  reduce  aircraft  availability 
rates  and  sortie  generation  rates.  Poor  accuracy  of  diagnostic  tools  can  also  lead  to 
desensitizing  maintenance  personnel  to  actual  faults  (The  Office  of  The  Director,  2015). 

Because  the  F-35’s  prognostic  capability  is  not  yet  functional,  only  the  diagnostic 
capability  is  modeled  in  this  research.  The  data  in  Table  5  are  used  as  the  baseline  values 
to  model  the  diagnostic  process  within  the  sortie  generation  process. 
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Learning  Curves  and  Reliability  Growth 

Organizations  and  their  workers  tend  to  operate  more  efficiently  over  time,  if  they 
perform  a  task  repetitively.  Learning  Curves  (LCs)  were  originally  proposed  by  Theodore 
Paul  Wright  in  1936  upon  observing  cost  reductions  due  to  repetitive  procedures  in 
aircraft  production  plants  (Anzanello  &  Fogliatto,  2011).  Figure  5  shows  that  it  takes  less 
cost  or  time  to  complete  each  additional  unit  as  the  number  of  repetitions  (volume) 
increases. 
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Figure  5.  Learning  Curve 

Since  their  first  introduction,  LCs  have  been  widely  applied  to  services  and 
industry.  LC  effects  within  some  major  U.S.  industries  are  presented  in  Table  6. 
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Table  6.  Examples  of  Learning  Curves  Effects  (Cunningham,  1980) 


Example 

Improving  Parameter 

Cumulative 

Parameter 

Learning- 

Curve 

Slope 

(%) 

1 .  Model-T  Ford  production 

Price 

Units  produced 

86 

2.  Aircraft  assembly 

Direct  labor-hours  per  unit 

Units  produced 

80 

3.  Equipment  maintenance 

Average  time  to  replace  a 

Number  of  replacements 

76 

at  GE 

4.  Steel  production 

group  of  parts 

Production  worker  labor-hours 

Units  produced 

79 

5.  Integrated  circuits 

per  unit  produced 

Average  price  per  unit 

Units  produced 

72a 

6.  Hand-held  calculator 

Average  factory  selling  price 

Units  produced 

74 

7.  Disk  memory  drives 

Average  price  per  bit 

Number  of  bits 

76 

Many  approaches  are  used  to  model  LCs  mathematically.  Generally,  Wright’s 
model,  which  is  also  referred  as  the  “Log-linear  Model”,  is  viewed  as  the  first  formal  LC 
model.  It  has  the  following  mathematical  representation: 


y  =  T,xh  (1) 

Where: 

y  =  the  average  time  (or  cost)  per  unit  demanded  to  produce 
x  =  the  cumulative  number  of  units  produced 
Ti  —  the  time  (cost)  to  produce  the  first  unit 

Parameter  b  =  the  slope  of  the  LC  which  describes  the  workers’  learning  rate. 

=  (log  of  the  learning  rate)  /  (log  2). 

Parameter  b  has  values  between  -1  and  1.  Values  of  b  close  to  -1  denote  high 
learning  rate  and  fast  adaptation  to  task  execution  (Anzanello  &  Fogliatto,  2011). 
According  to  the  model,  as  the  cumulative  number  of  the  output  is  doubled,  the  average 
time  (cost)  per  unit  decreases  by  b  percent. 
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As  stated  in  previous  sections,  the  sortie  generation  process  is  a  combination  of 
multiple  repetitive  tasks.  Therefore,  it  is  possible  that  the  time  required  to  perform  these 
tasks  may  decrease  as  logistics  personnel  gain  more  experience  after  each  flight  mission. 
The  2015  DOT&E  report  notes  that  a  learning  curve  effect  is  likely  to  improve  the  F-35’s 
repair  times.  As  maintainers  become  more  familiar  with  common  failure  modes,  their 
ability  to  quickly  repair  them  improves  over  time  (The  Office  of  The  Director,  2015). 

Another  important  implication  of  the  LCs  is  the  reliability  growth.  Complex 
systems  under  development  typically  face  high  initial  failure  rates.  However,  over  time  a 
learning  curve  effect  takes  place  as  sources  of  failures  are  determined  and  eliminated. 
Therefore,  the  failure  rates  start  to  gradually  decrease  (Jewell,  1984). 

The  Duane  model  is  one  of  the  most  common  reliability  growth  patterns 
experienced  in  practice  (Larry  H  .  Crow,  2011).  DOD  also  uses  the  Duane  model  to 
investigate  the  reliability  growth  for  aircraft.  Mean  Flight  Hours  Between  Unscheduled 
Maintenance  (MHFBSME)  growth  rates  calculated  by  the  DOD  for  several  historical 
aircraft  are  shown  in  Table  7  (The  Office  of  The  Director,  2015). 

Table  7.  Growth  Rates  for  Several  Historical  Aircraft  (The  Office  of  The  Director,  2015) 


Aircraft 

MFHBME 
Growth  Rate 

F-15 

0.14 

F-16 

0.14 

F-22  (at  35,000  flight  hours) 

0.22 

B-1 

0.13 

"Early" B-2  (at  5,000  flight  hours) 

0.24 

"Late"B-2 

0.13 

C-1 7  (at  1 5,000  flight  hours) 

0.35 
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However,  due  to  lack  of  the  data  to  implement  the  Duane  model,  this  research 
uses  the  Idealized  Growth  Curve  Model  which  is  a  simpler  reliability  growth  model 
defined  in  the  Military  Handbook  of  Reliability  Growth  Management  (United  States 
Department  of  Defense,  1981). 

According  to  the  Idealized  Growth  Curve  Model,  a  reliability  growth  rate  is 
calculated  through  the  equation  below. 


a  =  ln(B~1  +  {  I1  +  in£)]2  +  2<"Of 


(2) 


Where: 

a  =  Growth  parameter 

T  =  Cumulative  test  time  at  the  end  of  the  test 

ti  =  Length  of  initial  test  cycle  in  cumulative  test  time 

Mp  =  Final  Mean  Time  to  Failure  (MTTF) 

Mi  =  Initial  MTTF 

The  growth  rate  is  a  value  between  0  and  1.  Zero  means  no  growth.  As  the  growth 
rate  increases,  the  failure  rate  decreases.  Based  on  the  growth  rate  calculated  in  Equation 
2,  instantaneous  MTTF  is  obtained  from  equation  3. 


M(t)  — 


Mlt  Octets 


Mi 


cf)B.  t  >  t! 


Where: 

M(t)  =  Instantaneous  MTTF  at  time  t 
t  =  Cumulative  test  time 


(3) 
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M\  =  Average  MTTF  at  time  t\ 

t\  =  Length  of  initial  test  cycle  in  cumulative  test  time 

a  =  Growth  parameter 

Due  to  their  potential  impacts  mentioned  above,  LCs  and  reliability  growth 
concepts  are  addressed  while  developing  the  simulation  model.  Necessary  mathematical 
functions  are  embedded  into  the  model  accordingly.  Thus,  the  model  is  able  to  produce 
more  realistic  outputs  for  the  overall  sortie  generation  process.  Implementations  of  LCs 
and  reliability  growth  are  explained  thoroughly  in  the  methodology  section. 

Conclusion 

The  literature  reviews  indicated  that  many  valuable  studies  were  conducted  to 
investigate  the  sortie  generation  process  and  the  potential  impacts  of  the  AL  system  on 
the  sortie  generation  MOPs.  However,  they  mainly  examined  the  AL  system  on  a 
conceptual  basis,  since  real  world  F-35  data  was  not  yet  available.  Moreover,  none  of  the 
researches  incorporated  possible  impacts  of  the  learning  curves. 

Incorporating  the  learning  curves  and  using  the  most  recent  real  world  data  about 
the  AL  system,  this  research  builds  a  discrete  event  simulation  model  of  the  F-35’s  sortie 
generation  processes  in  order  to  provide  valuable  information  for  decision  makers. 
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III.  Methodology 


Chapter  Overview 

This  chapter  describes  a  discrete  event  simulation  model  of  the  F-35’s  sortie 
generation  process  under  the  Autonomic  Logistics  (AL)  system.  The  following  sections 
cover  data  collection,  definition  of  assumptions,  modeling  steps,  and  implementation  of 
Learning  Curves  (LCs)  and  reliability  growth  concepts. 

Simulation  in  Arena 

Sortie  generation  is  a  very  complicated  process  with  many  sub-processes  and 
numerous  decision  nodes.  Performing  an  analytical  analysis  may  be  extremely  time 
consuming,  challenging,  and  even  impossible  in  some  occasions.  At  this  point,  computer 
based  simulation  tools  provide  great  benefits  to  the  modelers. 

Arena®  simulation  software  is  used  in  this  research  to  model  the  F-35’  sortie 
generation  process  under  a  simplified  AL  system  that  approximates  current  ALIS 
capabilities.  It  is  a  flexible  and  powerful  tool  that  allows  analysts  to  create  animated 
simulation  models  that  accurately  represent  virtually  any  system  (Takus  &  Profozich, 
1997).  A  detailed  description  of  the  modeling  process  is  presented  in  the  following 
sections. 

Assumptions 

For  convenience,  some  assumptions  have  been  made  during  the  model  building 
stage.  Due  to  the  difficulty  of  demonstrating  every  activity  within  the  F-35’s  sortie 
generation  process,  it  is  assumed  that  a  sortie  generation  process  consists  only  of  the  sub- 
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processes  represented  in  Figure  1  and  that  excluded  activities  do  not  make  a  significant 
difference. 

The  F-35  aircraft  is  a  complex  weapon  system  including  a  large  number  of  Line 
Replaceable  Components  (LRC).  Its  mission  capability  depends  on  the  full  functionality 
of  all  LRCs.  However,  from  a  modelling  view,  it  is  not  feasible  to  model  every  LRC  and 
failure  type.  For  simplicity,  each  F-35  is  assumed  as  a  one-LRC  system  which  encounters 
only  mission  critical  failures  depending  on  the  accumulated  flight  hours.  Also,  scheduled 
maintenance  and  depot  level  maintenance  are  assumed  to  be  conducted  at  predetermined 
intervals  based  on  accumulated  flight  hours. 

As  stated  in  Chapter  2,  ALIS’s  diagnostic  capability  is  functional  now,  but  the 
prognostic  capability  is  not  yet  functional.  Therefore,  the  prognostic  capability  is  not 
included  in  the  model.  Also,  analyzing  the  impact  of  resource  levels  on  sortie  generation 
is  beyond  the  scope  of  this  research.  Therefore,  resource  capacity  is  assumed  to  be 
infinite  for  all  processes.  Process  modules  are  used  to  delay  the  aircraft  for  predetermined 
time  durations.  Meanwhile,  no  resource  is  seized  nor  released  by  the  aircraft.  Thus, 
queuing  problems  do  not  occur. 

DOD  reports  emphasize  that  there  will  be  a  learning  curve  and  reliability  growth 
effect  on  F-35’s  maintenance  processes.  However,  learning  rates  or  reliability  growth 
rates  and  the  processes  that  will  be  influenced  by  them  are  not  explicitly  known.  It  is 
assumed  that  while  LCs  have  impacts  on  the  process  times  of  the  Action  Request  (AR) 
system  and  maintenance  activities,  reliability  growth  influences  the  Mean  Flight  Hours 
Between  Critical  Failures  (MFHBCF). 
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Additional  assumptions  include:  a)  The  input  data  obtained  from  literature 
reviews  and  field  experts  adequately  reflect  real  operations;  b)  A  typical  flight  day  is  24 
hours  and  flight  year  is  365  days;  c)  Flight  missions  are  planned  at  a  constant  rate  during 
an  assumed  five-year  period;  d)  There  is  no  aircraft  loss  or  accident  in  flight  due  to  a 
failure;  e)  Mission  critical  failure  is  the  only  failure  type  encountered  by  the  F-35;  f) 
Probability  of  a  mission  critical  failure  during  preflight  inspection  and  final  check  is  0.05 
and  0.01  respectively;  g)  Probability  for  running  out  of  LRC  supply  is  0.50;  h)  For  85% 
of  the  mission  critical  failures,  a  known  fix  is  available  in  the  Joint  Technical  Documents; 
i)  For  50%  of  the  time,  maintainers  are  able  to  detect  PHM-related  false  alarms  during  the 
troubleshooting  process;  j)  The  aircraft  in  the  model  is  the  F-35’s  conventional  variant. 

Data  Collection 

Data  collection  is  an  important  simulation  step  and  directly  affects  model  validity. 
Since  the  F-35  is  a  relatively  new  system,  it  was  difficult  to  obtain  actual  data  for  all 
processes  within  the  model.  Therefore,  when  actual  data  were  not  available,  data  sets 
belonging  to  other  fighter  aircraft  were  collected  and  used  to  model  the  associated 
processes. 

The  majority  of  process  delay  times  were  taken  from  the  “US  Air  Force 
Maintenance  Capability  and  Capacity  Modeling  and  Simulation  Summary  Technical 
Report”  (Spencer,  Hall,  &  Ostrander,  2010).  Other  process  times  were  obtained  from 
previous  studies  visited  during  the  literature  reviews  (Faas,  2003;  Rossetti  &  McGee, 
2006;  Sheppard,  2014).  Delay  times  for  the  activities  in  the  sortie  generation  process  and 
their  related  statistical  distributions  are  presented  in  Table  8. 
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Table  8.  Process  Times  and  Related  Distributions 


Process 

Time  and  Distribution 

Reference 

Refuel 

*Normal  (0.5,  0.145)  hours 

(Spencer,  Hall,  & 

Ostrander,  2010b) 

Other  Servicing  (Oil,  Liquid 
oxygen,  Hydraulics,  or  Tires) 

Normal  (0.3,  0.087)  hours 

(Spencer  et  al.,  2010b) 

Configuration  (Weapon 
Loading  or  Pod  Installation) 

Uniform  (28,  249)  minutes 

Adjusted  from  (Spencer  et 
al.,  2010b) 

Pre-Flight  Inspection 

Triangular  (50,60,70)  minutes 

(Faas,  2003) 

Engine  Start,  Final  Systems 
Check,  and  Taxiing 

Normal  (0.8,  0.232)  hours 

(Spencer  et  al.,  2010b) 

Takeoff 

Triangular  (2,3,4)  minutes 

(Faas,  2003) 

Sortie 

Normal  (2,  0.5)  hours 

(Faas,  2003) 

Landing 

Triangular  (14,15,16)  minutes 

(Faas,  2003) 

Parking  and  Recovery 

Triangular  (5,7,9)  minutes 

(Faas,  2003) 

Downloading  PMD  into 

ALIS 

Triangular  (7,10,13)  minutes 

Adjusted  from  (The  Office 
of  The  Director,  2015) 

Basic  Post-Flight  Operations 
and  Aircrew  Debrief 

Normal  (2,  0.58)  hours 

(Spencer  et  al.,  2010b) 

Troubleshooting 

Triangular  (20,  24,  30)  minutes 

(Faas,  2003) 

Wait  for  Part  to  issue  from 
supply 

Triangular  (0.5,  2,  2.5)  hours 

(Rossetti  &  McGee,  2006) 

Unscheduled  Maintenance 

Triangular  (9,  9.7,  10.4)  hours 

Adjusted  from  (The  Office 
of  The  Director,  2015) 

Scheduled  Maintenance 

Triangular  (5,7,8)  days 

(Rossetti  &  McGee,  2006) 

Depot  Level  Maintenance 

Triangular  (110,  131,  144)  days 

(Sheppard,  2014) 

*Normal  (Mean,  Standard  Deviation) 


The  F-35A’s  maintenance  data  and  PHM  statistics  were  obtained  from  the  2015 
DOT&E  report  and  are  presented  in  Table  9.  According  to  this  data,  the  F-35  PHM 
system  has  the  capability  of  detecting  84%  of  the  mission  critical  failures  with  85%  of 
those  detections  being  correct.  Furthermore,  in  the  instance  of  the  failure  being  a  non¬ 
electronic  fault,  79%  of  the  correct  detections  are  isolated  successfully  to  a  single  FRC. 
Therefore,  the  PHM  system  can  accurately  detect  only  71.4%  of  the  mission  critical  non¬ 
electrical  failures  and  successfully  isolate  only  57.1%  of  them  to  a  FRC. 
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Table  9.  Actual  MFHBCF,  MCMTCF,  MFHBFA  and  PHM  Data  for  F-35A  (The  Office  of  The 

Director,  2015) 


Mean  Flight  Hours  Between  Critical  Failures  (MFHBCF) 

10.2  hours 

Mean  Corrective  Maintenance  Time  for  Critical  Failure  (MCMTCF) 

9.7  hours 

Mean  Flight  Hours  Between  Flight  Safety  Critical  False  Alarm 

170  hours 

Fault  Detection  Coverage  (percent  mission  critical  failures  detectable  by 

PHM) 

84% 

Fault  Detection  Rate  (percent  correct  detections  for  detectable  failures) 

85% 

Fault  Isolation  Rate  (percent  isolation  of  Non-Electronic  Fault  to  One  LRC 

79% 

The  AR  system  within  the  ALIS  is  used  to  provide  resolutions  to  the  problems  of 
which  known  fixes  are  not  available  in  the  technical  documents  or  PHM  system.  A  2016 
study  conducted  by  Colbacchini  et  al.  (2016)  provided  some  valuable  data  about  the 
resolution  process  of  Category  1  (critical)  problems  through  the  AR.  Our  research 
translates  their  dataset  into  triangular  distributions  used  to  model  the  AR  system  as 
presented  in  Table  10. 


Table  10.  Process  Times  within  the  AR  System 


Initiation  Process 

Triangular  (12,  24,  30)  hours 

Optional  Screening  Point  (OSP) 

Triangular  (12,  24,  30)  hours 

Required  Screening  Point  (RSP) 

Triangular  (1,2,  2.5)  days 

Resolution 

Triangular  (9,  11,  13)  days 

Model  Development 

Our  simulation  model  represents  the  sortie  generation  process  of  a  16-aircraft  F- 
35  fleet  in  a  notional  base  in  Turkey  over  a  five-year  period.  There  are  24  hours  in  a 
flight  day  and  365  days  in  a  flight  year.  Flight  missions  are  planned  at  a  constant  rate.  If 
there  is  an  available  aircraft  in  the  aircraft  pool,  the  mission  is  initiated  immediately. 
Otherwise,  the  mission  is  cancelled. 
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The  global  input  variables  used  in  the  model  are  presented  in  Table  11. 


MFHBFA,  fault  coverage  rate,  correct  fault  detection  rate,  fault  isolation  rate,  reliability 
growth  rate,  and  learning  curve  rate  are  determined  as  the  critical  factors  which  are  used 
during  the  design  of  experiment  (DOE)  stage.  Their  impacts  on  the  sortie  generation 
process  are  examined  by  setting  them  to  different  levels  under  different  scenarios. 


Table  11.  Global  Variables 


Variable 

Initial  value 

Unit 

*MFHBFA 

170 

HOUR 

MFHBCF 

10.2 

HOUR 

MFHBDFME 

2,000 

HOUR 

MFHBSME 

400 

HOUR 

FFIGHT  TIME 

0 

HOUR 

*FAULT  COVERAGE  RATE 

84 

PERCENT 

*CORRECT  FAULT  DETECTION  RATE 

85 

PERCENT 

*FAULT  ISOLATION  RATE 

79 

PERCENT 

HEALTHY 1 

95 

PERCENT 

HEALTHY2 

99 

PERCENT 

AR  CORRECT 

95 

PERCENT 

*RELIABILTY  GROWTH  RATE 

0 

PERCENT 

*LEARNING  CURVE  RATE 

100 

PERCENT 

PLANNED  FLIGHT  HOURS 

0 

HOUR 

FLYING  SCHEDULING  EFFECTIVENESS  RATE 

0 

PERCENT 

AIRCRAFT  AVAILABILITY  RATE 

0 

PERCENT 

SUPPLY  AVAILABILITY 

50 

PERCENT 

*  KNOWN  FIX  AVAILABILITY 

85 

PERCENT 

*Critical  factors  used  in  the  DOE 


An  overall  view  of  the  simulation  model  is  presented  in  Figure  6.  It  was  built 
based  on  the  sortie  generation  activities  defined  in  Figure  1.  The  model  building  process 
is  explained  thoroughly  in  the  following  sections. 
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Figure  6.  Sortie  Generation  Model 


Model  Initialization 

The  simulation  model  is  started  with  the  creation  of  16  aircraft  entities  which  are 
then  routed  to  the  aircraft  pool  to  wait  for  a  flight  mission  signal  from  the  mission 
planning  area.  The  Aircraft  pool  is  a  hold  module  which  releases  one  aircraft  after  a 
mission  signal  is  received  from  the  flight  planning  area.  After  release,  the  flight  duration 
determined  in  the  flight  planning  area  is  assigned  to  the  aircraft,  and  the  aircraft  goes  to 
the  preflight  operations  area.  On  completion  of  the  sortie  generation  cycle,  the  aircraft 
does  not  leave  the  model;  it  returns  to  the  aircraft  pool  and  waits  for  the  next  mission. 

The  flight  scheduling  area  is  a  combination  of  create,  assign,  signal,  record,  and 
disposal  modules.  A  flight  mission  is  created  every  hour  at  a  constant  rate.  Once  a  flight 
mission  is  created,  an  assign  module  determines  the  flight  duration  according  to  a  normal 
distribution  with  a  mean  of  2  hours  and  standard  deviation  of  0.  5  hours.  Then,  a  mission 
signal  is  sent  to  the  hold  module  to  release  an  aircraft  from  the  aircraft  pool.  If  any 
aircraft  is  available,  the  flight  mission  starts.  Otherwise,  the  flight  is  cancelled.  After  the 
mission  is  signaled,  an  assign  module  accumulates  the  planned  flight  hours  and  assigns  it 
to  PLANNED  FLIGHT  HOURS  global  variable.  An  overall  view  of  the  model 
initialization  is  presented  in  Figure  7. 
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Figure  7.  Model  Initialization  Process 


Preflight  Operations 

After  an  aircraft  is  released  from  the  aircraft  pool,  it  is  routed  to  the  refueling 
area.  Then,  a  decide  module  checks  the  need  for  additional  servicing.  Fifty  percent  of  the 
time,  the  aircraft  requires  servicing  for  oil,  liquid  oxygen,  hydraulics,  or  tire  check. 
Following  the  servicing,  a  transition  from  one  configuration  to  another  starts.  It  involves 
alternate  mission  equipment  download  and  upload  and  munitions  upload.  After  the 
aircraft  is  configured,  a  pre-flight  inspection  is  conducted  to  check  if  any  mission  critical 
failure  is  present  or  not.  During  the  inspection,  there  is  a  0.05  probability  that  a  mission 
critical  failure  is  detected.  If  there  is  a  failure,  the  aircraft  is  routed  to  the  PHM  area; 
otherwise  it  is  transferred  to  the  sortie  area.  The  overall  view  of  the  pre-flight  operations 
is  presented  in  Figure  8. 


Figure  8.  Pre-flight  Operations 
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Sortie 


Completing  the  pre-flight  operations,  the  engine  is  started  and  final  checks  are 
conducted  to  detect  possible  mission  critical  failures.  At  this  point,  the  probability  of  a 
mission  critical  failure’s  occurrence  is  0.01.  If  there  is  a  failure,  the  aircraft  is  routed  to 
the  PHM  area;  otherwise  it  releases  the  parking  area  and  begins  taxiing  onto  the  runway 
for  take-off.  The  aircraft  takes  off  and  the  sortie  is  executed  based  on  a  flight  duration 
predetermined  in  the  mission  planning  area.  Then  the  aircraft  comes  to  the  landing 
module  and  is  delayed  there.  The  overall  view  of  the  sortie  is  presented  in  Figure  9. 


Figure  9.  Sortie  Process 


Following  the  landing,  an  assign  module  calculates  the  flying  scheduling 
effectiveness  (FSE)  rate  of  the  fleet  and  assigns  it  to  the  FLYING  SCHEDULING 
EFFECTIVENESS  RATE  global  variable.  Additionally,  the  attributes  presented  below 
are  assigned  to  the  aircraft.  These  attributes  are  flight  hours  accumulated  by  each  aircraft. 
Their  usages  are  explained  in  the  following  sections. 

•  AIRCRAFT  MFHBCF:  Flight  Hours  Between  Critical  Failures 

•  AIRCRAFT  MFHBSME:  Flight  Hours  Between  Scheduled  Maintenance 

•  AIRCRAFT  MFHBFA:  Flight  Hours  Between  False  Alarms 

•  AIRCRAFT  MFHBDLME:  Flight  Hours  Between  Depot  Level 
Maintenance 
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Post-Flight  Operations 

After  landing,  the  aircraft  is  routed  to  the  parking  area.  Then,  aircrew  debriefing 
and  basic  post-flight  operations  (BPO)  are  conducted  concurrently.  During  BPO, 
maintainers  download  post  flight  Health  Reporting  Codes  (HRC)  from  the  aircraft  to 
ALIS  through  a  Ground  Data  Security  Assembly  Receptacle  (GDR).  The  HRCs  are  used 
by  ALIS’s  PHM  system  to  check  the  aircraft  health  status. 

PHM  Area 


Holding  the  fault  coverage,  fault  detection,  fault  isolation  and  false  alarm  logic, 
the  PHM  area  is  the  most  crucial  part  of  the  model.  It  consists  of  a  combination  of  assign 
and  decision  modules.  The  overall  view  of  the  area  is  represented  in  Figure  10. 


Figure  10.  PHM  Area 


At  the  beginning  of  the  PHM  process,  a  decide  module  checks  for  false  alarms.  A 
false  alarm  (false  positive)  indicates  that  there  is  a  failure  given  that  none  exists.  The 
decide  module  compares  the  AIRCRAFT  MFHBFA  attribute  value  to  a  predetermined 
MFHBFA  (170  hours)  to  decide  whether  there  is  a  false  alarm  or  not.  If  the  related 
aircraft’s  AIRCRAFT  MFHBFA  attribute  value  exceeds  the  MFHBFA  value,  then  a  false 
alarm  occurs.  If  there  is  a  false  alarm,  then  the  value  one  is  assigned  to  the  AIRCRAFT 
FALSE  ALARM  attribute  value;  the  AIRCRAFT  MFHBFA  attribute  value  is  reset  to 
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zero  and  the  aircraft  is  sent  to  a  decide  module  checking  the  fleet  cumulative  flight  hours. 
Otherwise,  the  aircraft  is  directly  sent  to  the  decide  module  checking  the  fleet  cumulative 
flight  hour. 

Next,  a  decide  module  checks  whether  the  fleet  cumulative  flight  hours  are  above 
or  below  the  2100-hour  level,  which  is  the  starting  point  of  reliability  growth  for  Mean 
Flight  Hours  Between  Critical  Failures  (MFHBCF).  As  presented  in  Table  8,  currently 
the  MFHBCF  is  10.2  hours.  However,  after  the  fleet  exceeds  2100  flight  hours,  it  begins 
to  improve.  A  detailed  explanation  for  the  implementation  of  the  reliability  growth  is 
presented  in  the  following  sections. 

After  the  reliability  growth  check,  an  unscheduled-maintenance  check  module 
compares  MFHBCF  to  the  AIRCRAFT  MFHBCF  attribute  value  and  determines  the 
current  condition  of  the  aircraft.  At  this  point,  an  aircraft  may  be  in  one  of  the  following 
categories:  A)  Critical  failure,  B)  Critical  failure  and  false  alarm  (treated  as  critical 
failure),  C)  False  alarm,  D)  Healthy.  Category  A  means  that  the  AIRCRAFT  MFHBCF 
attribute  value  exceeded  the  MFHBCF  and  a  mission  critical  failure  occurred.  Category  B 
means  that  both  a  critical  failure  occurred  and  a  false  alarm  was  assigned  to  the  aircraft 
previously  in  the  false  alarm  check  area.  Category  C  means  that  no  critical  failure 
occurred  but  a  false  alarm  was  assigned  to  the  aircraft  in  the  previous  module.  Category 
D  means  that  the  aircraft  has  neither  a  critical  failure  nor  a  false  alarm. 

If  the  aircraft  is  in  category  A  or  B,  it  is  sent  to  an  assign  module  and  value  of 
one  is  assigned  to  the  AIRCRAFT  REAL  ALARM  attribute  value,  and  then  the  aircraft  is 
transferred  to  the  failure  coverage  check.  If  the  aircraft  is  in  category  C,  it  is  directly  sent 
to  the  failure  coverage  check  without  being  assigned  a  real  alarm.  Linally,  if  the  aircraft  is 
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in  category  D,  it  skips  the  unscheduled  maintenance  and  directly  goes  to  the  scheduled 
maintenance  check. 

If  there  is  a  false  alarm  or  real  failure,  a  fault  coverage  check  is  applied  to  see 
whether  the  fault  is  covered  by  the  PHM  or  not.  The  current  PHM  fault  coverage  rate  is 
0.84.  If  the  failure  is  not  within  the  PHM  coverage,  the  aircraft  is  routed  to  the 
troubleshooting  area  for  further  inspections;  otherwise  it  is  transferred  to  another  deicide 
module  to  see  whether  the  fault  detection  is  correct  or  not.  The  current  correct  detection 
rate  of  the  PHM  system  is  0.85.  If  the  detection  is  not  correct,  an  assign  module  assigns 
value  of  one  to  attribute  MISDETECTION,  and  then  the  aircraft  is  routed  to  the  fault 
isolation  check.  Otherwise,  the  aircraft  is  directly  sent  to  the  fault  isolation  check  module. 
Misdetection  means  that  the  aircraft  has  a  problem,  but  it  is  defined  inaccurately; 
therefore,  an  incorrect  maintenance  action  would  be  applied  to  attempt  to  fix  it. 

After  fault  detection,  the  fault  isolation  module  checks  whether  the  fault  can  be 
isolated  to  one  specific  LRC  or  not.  The  current  fault  isolation  rate  of  the  PHM  system  is 
0.79.  If  the  fault  can  be  isolated  to  one  specific  LRC,  then  the  AIRCRAFT  REAL 
ALARM  and  the  AIRCRAFT  FALSE  ALARM  attribute  values  are  reset  to  zero  and  the 
aircraft  is  routed  to  the  unscheduled  maintenance  module;  otherwise  the  aircraft  is  sent  to 
the  troubleshooting  area  for  further  inspection. 

Troubleshooting  and  Action  Request 

If  a  mission  critical  failure  is  not  covered  or  isolated  by  the  PHM  system,  then  the 
aircraft  is  routed  to  a  troubleshooting  process  for  further  inspection.  After  the 
troubleshooting,  a  decide  module  checks  whether  the  failure  is  a  real  alarm  or  not.  If  it  is 
a  real  alarm,  then  the  AIRCRAFT  REAL  ALARM  attribute  value  is  reset  to  zero  and  the 
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aircraft  is  transferred  to  the  known  fix  availability  check.  If  it  is  a  false  alarm,  the 
AIRCRAFT  FALSE  ALARM  attribute  value  is  reset  to  zero  and  a  decide  module  checks 
whether  maintainers  can  detect  the  false  alarm  or  not.  It  is  assumed  that  maintainers  catch 
a  false  alarm  50  percent  of  the  time.  If  the  maintainers  catch  the  false  alarm,  the  aircraft 
skips  the  unnecessary  unscheduled  maintenance  and  directly  goes  to  a  scheduled 
maintenance  check.  Otherwise,  the  aircraft  is  routed  to  the  known  fix  availability  check 
and  ends  with  an  unnecessary  unscheduled  maintenance  action. 

After  the  real  and  false  alarm  checks,  a  decide  module  checks  whether  a  known 
fix  is  available  in  the  technical  documents  or  not.  The  probability  of  a  known  fix  is  0.85. 
If  the  known  fix  is  available,  the  aircraft  is  directly  sent  to  the  unscheduled  maintenance 
area;  otherwise  an  action  request  (AR)  is  initiated  to  find  a  solution  to  the  fault  by  the 
assistance  of  Lockheed  Martin  engineers.  Overall  views  of  the  troubleshooting  process 
and  the  AR  system  are  represented  in  Ligure  1 1  and  Ligure  12. 


Figure  11.  Troubleshooting  Process 
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An  AR  is  delayed  within  the  AR  system  according  to  the  times  and  distributions 
presented  in  Table  10.  OPS  and  RSP  checkpoints  ensure  that  the  AR  is  detailed  and 
complete  before  sending  the  AR  to  the  Lockheed  Martin  engineers.  There  is  a  0.95 
probability  that  AR  passes  through  these  checkpoints  without  any  problem.  If  there  is  a 
problem,  the  AR  is  sent  back  to  the  initiation  step  (Colbacchini  et  al.,  2016).  After  a 
resolution  is  found,  the  aircraft  is  routed  to  the  unscheduled  area. 

Colbacchini  et  al.  (2016)  indicated  that  current  AR  process  times  are  longer  than 
the  desired  values.  Our  research  assumes  that  a  learning  curve  effect  takes  place  within 
the  AR  process.  A  detailed  explanation  is  presented  in  the  Implementation  of  the 
Learning  Curves  section. 

Maintenance  Operations 

Maintenance  processes  consist  of  unscheduled  maintenance,  scheduled 
maintenance,  and  depot  level  maintenance.  Figure  13  represents  the  overall  maintenance 
area. 
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Figure  13.  Maintenance  Processes 


Once  the  aircraft  is  routed  to  the  unscheduled  maintenance  area,  a  record  module 
counts  the  number  of  unscheduled  maintenance.  Before  the  maintenance,  a  decide 
module  checks  if  there  are  an  adequate  number  of  LRCs  or  not.  Fifty  percent  of  the  time, 
the  LRC  quantity  is  insufficient  and  the  aircraft  is  delayed  until  replenishment  arrives.  If 
there  is  enough  supply,  the  unscheduled  maintenance  is  initiated  immediately.  After  the 
maintenance  is  conducted,  the  AIRCRAFT  MFHBCF  attribute  value  is  reset  to  zero  and 
the  aircraft  is  sent  to  a  decide  module  for  functionality  check.  If  the  MISDETECTION 
attribute  value  is  one,  then  previous  fault  detection  and  fault  isolation  actions  were  done 
inaccurately  by  the  PHM  system  and  the  wrong  maintenance  activity  was  conducted  to 
the  aircraft.  Therefore,  the  MISDETECTION  attribute  value  is  reset  to  zero  and  the 
aircraft  is  routed  to  the  troubleshooting  module  for  further  inspection. 

If  the  MISDETECTION  attribute  value  is  zero,  the  aircraft  is  routed  to  another 
decide  module  for  a  scheduled  maintenance  check.  Scheduled  maintenance  is  conducted 
based  on  the  accumulated  flight  hours.  The  default  value  for  mean  flight  hours  between 
scheduled  maintenance  is  400  hours  and  is  recorded  in  the  MFHBSME  global  variable.  If 
the  AIRCRAFT  MFHBSME  attribute  value  does  not  exceed  the  MFHBSME  variable 
value,  the  aircraft  is  directly  sent  to  depot  level  maintenance;  otherwise  scheduled 
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maintenance  is  conducted.  After  the  scheduled  maintenance,  the  AIRCRAFT 


MFHBSME  is  reset  to  zero  and  the  aircraft  is  routed  to  depot  level  maintenance. 

When  the  aircraft  needs  any  modification  or  upgrade,  it  is  sent  to  the  depot  level 
maintenance  area.  Like  scheduled  maintenance,  depot  level  maintenance  is  also 
determined  according  to  the  accumulated  flight  hours.  The  default  value  of  mean  flight 
hours  between  depot  level  maintenance  is  2000  hours  and  recorded  in  the  MFHBDLME 
global  variable.  If  the  AIRCRAFT  MFHBDLME  attribute  value  does  not  exceed  the 
MFHBDLME  variable  value,  then  the  aircraft  is  routed  directly  to  the  aircraft  pool; 
otherwise  the  aircraft  undergoes  depot  level  maintenance.  After  the  depot  level 
maintenance,  the  AIRCRAFT  MFHBD  is  reset  to  zero  and  the  aircraft  is  transferred  to 
the  aircraft  pool. 

Implementation  of  the  Reliability  Growth  for  the  MFHBCF 

The  2015  DOT&E  report  indicates  that  reliability  growth  takes  place  within  the  F- 
35  system.  As  shown  in  Table  12,  while  MFHBCF  value  for  F-35A  was  8.2  hours  at 
8,834  cumulative  flight  hours,  it  improved  to  10.2  hours  at  15,845  cumulative  flight 
hours.  The  target  value  for  MFHBCF  at  75,000  cumulative  flight  hours  was  defined  as  20 
hours. 


Table  12.  Cumulative  Flight  Hours  versus  MFHBCF 


Cumulative  Flight  Hours 

MFHBCF 

8,834  (August  2014) 

8.2 

15,845  (May  31  2015) 

10.2 

75,000  (Maturity  level) 

20 
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According  to  Table  12,  in  a  10-month  period  between  August  2014  and  May  31 
2015,  nearly  7,000  flight  hours  were  accumulated  by  the  F-35A,  which  equates  to  700 
flight  hours  per  month.  The  most  recent  MFHBCF,  10.2  hours,  was  calculated  based  on 
the  flight  hours  accumulated  within  a  three-month  rolling  window  starting  in  March  1 
2015  and  ending  in  May  31  2015.  Therefore,  it  can  be  derived  that,  by  March  1  2015, 
accumulated  flight  hours  were  approximately  13,745  and  the  remaining  flight  hours  to 
maturity  (75,000  flight  hours)  were  about  61,000  hours. 

If  the  March  1  2015  is  assumed  as  the  initial  point  for  a  reliability  test,  we  obtain 
the  parameter  values: 

Cumulative  test  time  at  the  end  of  the  test  (T)  =  6,100  flight  hours 
Length  of  initial  test  cycle  in  cumulative  test  time  (t\)  =  2,100  flight  hours 
Final  Mean  Time  To  Failure  (MTTF)  (Mr)  =  20  flight  hours 
Initial  MTTF  (Mi)  =  10.2  flight  hours 

Using  the  data  above,  a  reliability  growth  rate  can  be  calculated  using  Equation  2. 
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Anticipated  MFHBCF  values  are  presented  in  Table  13  for  some  cumulative  flight  hour 


levels  when  the  growth  rate  is  assumed  to  be  0.15. 


Table  13.  Reliability  Growth  for  MFHBCF 


MFHBCF 

Cumulative  Flight  Hours 

12.7 

3,000 

13.8 

5,000 

15.2 

10,000 

16.2 

15,000 

16.9 

20,000 

18.8 

40,000 

19.9 

60,000 

In  the  simulation  model,  reliability  growth  rate  was  defined  as  a  global  variable 
with  a  default  value  of  zero.  Equation  3  was  embedded  into  the  decide  module  that 
checks  the  unscheduled  maintenance  need  caused  by  the  mission  critical  failures.  During 
the  design  of  experiment  stage,  reliability  growth’s  impact  on  the  sortie  generation 
performance  measures  is  investigated  by  changing  the  growth  rate  under  different 
scenarios. 


Implementation  of  the  Learning  Curves 

Colbacchini  et  al.  (2016)  indicated  that  current  AR  process  times  are  longer  than 
the  desired  values.  These  delay  times  considerably  increase  the  aircraft  downtime,  since 
the  aircraft  is  in  a  non-mission  capable  state  until  a  resolution  is  found  for  the  AR. 
However,  Chapter  2  notes  that  it  is  possible  that  the  time  required  to  perform  AR  tasks 
may  decrease  as  maintenance  personnel  gain  more  experience  after  each  AR  initiation. 

Therefore,  Equation  1  is  embedded  into  the  related  AR  process  modules  to 
implement  a  learning  curve  (LCs)  effect  on  the  AR  activities.  Learning  rate  is  defined  as 
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a  global  variable  with  a  default  value  of  one  which  means  there  is  no  learning  from 
experience.  During  the  design  of  experiment  stage,  LCs’  impact  on  the  sortie  generation 
performance  measures  is  investigated  by  changing  the  learning  rate  under  different 
scenarios.  Assuming  a  95%  learning  rate,  possible  change  in  the  average  time  of  the 
resolution  process  is  presented  in  Table  14. 


Table  14.  Impact  of  a  95%  Learning  Rate  on  the  Resolution  Time 


Time  (days) 

Number  of  Repetitions 

13.00 

1 

12.35 

2 

11.98 

3 

11.15 

8 

10.64 

15 

10.11 

30 

9.25 

100 

8.21 

500 

The  second  place  for  a  potential  LC  effect  is  within  unscheduled  maintenance. 
Presently,  the  Mean  Corrective  Maintenance  Time  for  Critical  Failure  (MCMTCF)  is  9.7 
hours,  while  the  desired  value  is  4  hours.  The  2015  DOT&E  report  highlights  that 
learning  curve  effect  is  likely  to  improve  the  F-35’s  repair  times.  As  maintainers  become 
more  familiar  with  common  failure  modes,  their  ability  to  repair  them  more  quickly 
improves  over  time  (The  Office  of  The  Director,  2015).  Therefore,  a  second  LC  function 
is  embedded  into  the  unscheduled  maintenance  module.  Assuming  a  95%  learning  rate, 
possible  change  in  the  average  unscheduled  maintenance  process  time  is  presented  in 
Table  15. 
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Table  15.  Impact  of  the  95%  Learning  Rate  on  the  MCMTCF 


Time  (hours) 

Number  of  Repetitions 

9.70 

1 

9.22 

2 

8.18 

10 

7.26 

50 

6.12 

500 

5.82 

1,000 

5.53 

2,000 

5.25 

4,000 

4.99 

8,000 

Model  Verification  and  Validation 

Model  verification  and  validation  are  vital  steps  for  a  simulation  study  to  provide 
realistic  outputs.  Even  a  simple  structural  mistake  or  logic  error  may  cause  the  results  to 
dramatically  deviate  from  their  true  values.  To  build  the  model  correctly,  the  entire  sortie 
generation  process  was  divided  into  relatively  simpler  sub  processes  and  each  sub  process 
was  developed  individually.  Complexity  was  added  to  the  model  gradually  and  each 
major  addition  was  saved  as  a  different  version  to  avoid  potential  data  loss.  The  model 
was  animated  frequently  to  check  whether  the  aircraft  flowed  through  the  modules 
reasonably  or  not.  That  was  very  helpful  to  detect  the  errors  in  the  model  logic.  After  the 
model  was  complete,  it  was  run  for  30  replications  to  check  if  it  produced  a  flying 
scheduling  effectiveness  (FSE)  rate  similar  to  the  real  world  data  obtained  from  the  2015 
DOT&E  report,  when  the  aircraft  availability  (AA)  rate  was  close  to  the  real-world  data. 
Logically,  when  the  model  produced  AA  rates  close  to  real  world  data  (51%),  we 
expected  the  FSE  rate  also  to  be  close  to  the  real  world  data.  The  comparisons  in  the 
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Table  16  show  that  at  an  AA  rate  close  to  the  real  world  data,  our  model  produces  a  FSE 


rate  reasonably  close  to  the  real-world  data. 


Table  16.  Real  World  FSE  Rate  versus  Simulation  FSE  Rate 


Real  World  Data 

Simulation  Result 

AA  Rate  (%) 

FSE  Rate  (%) 

AA  Rate  (%) 

FSE  Rate  (%) 

51 

65 

49.47 

69.28 

For  the  model  validation,  three  subject  matter  experts  (SME)  were  consulted. 
Based  on  their  evaluations  and  suggestions,  some  important  changes  in  the  process  flows 
and  input  data  were  made  to  meet  the  operational  needs  of  the  air  force.  Moreover, 
official  ALS  reports  and  Air  Force  documents  were  used  to  ensure  an  acceptable  level  of 
model  validity.  Therefore,  it  was  decided  that  the  model  was  appropriate  for  the  needs  of 
the  Air  Force. 


Conclusion 

The  simulation  model  in  this  study  was  built  to  represent  the  sortie  generation 
process  of  an  F-35A  fleet  under  AES.  Actual  condition  of  the  AES  was  taken  into 
consideration  rather  than  the  theoretical  expectations,  since  ALIS  does  not  yet  meet  the 
desired  level  of  functionality.  During  the  model  building  stage,  the  main  attention  was 
given  to  the  PHM  system  and  the  AR  system  due  to  their  uniqueness  to  the  F-35.  Only 
mission  critical  failures  were  considered,  since  most  of  the  available  F-35  data  was 
related  to  them. 

Literature  reviews  indicated  that  little  prior  research  was  conducted  examining  the 
impact  of  the  learning  curves  and  reliability  growth  on  the  sortie  generation  process. 
However,  as  the  2015  DOT&E  report  indicated,  it  is  very  likely  that  the  F-35  will  be 
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influenced  by  these  concepts.  Therefore,  appropriate  LCs  and  reliability  growth  equations 
were  embedded  into  the  model. 

This  chapter  thoroughly  explained  the  data  collection,  model  building,  and 
implementation  of  the  LCs  and  reliability  growth.  Results  obtained  from  the  simulation 
model  and  their  related  analyses  are  presented  in  the  following  chapter. 
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IV.  Analysis  and  Results 


The  previous  chapter  explained  the  simulation  modeling  of  the  F-35’s  sortie 
generation  process.  This  chapter  covers  the  steps  which  are  followed  to  obtain  and 
analyze  the  simulation  results.  These  steps  include  defining  the  key  performance 
measures  of  the  sortie  generation  process,  building  a  designed  experiment,  and 
statistically  analyzing  the  experimental  outputs. 

Measures  of  Performance  (MOP)  for  the  Sortie  Generation  Process 

Literature  reviews  showed  that  aircraft  reliability,  aircraft  maintainability,  and 
aircraft  availability  are  some  critical  MOPs  used  to  evaluate  the  performance  of  an  air 
force  logistics  system. 

Aircraft  reliability  is  related  to  failure  frequency  encountered  by  the  aircraft.  Its 
assessment  includes  a  variety  of  metrics  like  Mean  Flight  Hours  Between  Critical  Failure, 
Mean  Flight  Hours  Between  Removal,  and  Mean  Flight  Hours  Between  Maintenance 
Events.  Each  metric  characterizes  a  unique  aspect  of  overall  weapon  system  reliability. 

Aircraft  maintainability  is  a  measure  to  assess  the  amount  of  the  time  needed  to 
repair  an  aircraft  to  return  it  to  flying  status  again.  Its  main  metric  is  the  Mean  Time  To 
Repair. 

Aircraft  availability  (AA)  rate  is  determined  by  measuring  the  percent  of  time 
that  an  individual  aircraft  is  in  the  “available”  status,  aggregated  over  a  reporting  period 
(The  Office  of  The  Director,  2015).  The  aircraft  which  are  not  available  are  assigned  to 
one  of  three  categories:  Not  Mission  Capable  for  Maintenance,  Not  Mission  Capable  for 
Supply,  or  Depot  status. 
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For  this  research,  AA  rate  and  flying  scheduling  effectiveness  (FSE)  rate  were 
chosen  as  the  key  MOPs  of  the  F-35’s  sortie  generation  process.  FSE  rate  is  calculated  by 
dividing  the  total  flight  hours  to  the  total  planned  flight  hours  and  it  is  directly  related  to 
the  AA  rate.  While  high  availability  rates  ensure  more  flight  hours  achievement,  low 
availability  rates  prevent  the  fleet  from  achieving  the  planned  flight  hour  goals. 

Run  Length  and  Replication 

The  simulation  run  length  was  determined  as  5  years  and  24  hours  a  day  to  allow 
each  aircraft  to  go  through  depot  level  maintenance  at  least  once.  Moreover,  this  run 
length  was  useful  to  see  the  impacts  of  the  learning  curves  and  reliability  growth  in  the 
long  term. 

After  deciding  the  simulation  run  length,  the  second  step  was  to  determine  the 
number  of  replications  adequate  to  obtain  accurate  results  from  the  simulation 
experiment.  Literature  reviews  showed  that  there  is  no  simple  guidance  on  the  number  of 
replicates  needed.  Although  more  replication  leads  to  more  successful  analysis,  cost  or 
time  considerations  often  dictate  the  number  of  replicates  that  can  be  achieved.  Previous 
simulation  studies  generally  used  20  to  30  replications.  For  this  research,  an  initial 
replication  number  was  selected  as  30  rather  than  20  since  the  additional  computation 
time  between  20  replications  and  30  replications  was  minimal. 

Next,  a  pilot  experiment  was  conducted  to  check  whether  30  replications  were 
enough  to  obtain  normally  distributed  output  data  or  not.  Then,  output  data  were 
imported  into  the  JMP®  Software,  and  Shapiro-Wilk  goodness  of  fit  (GOF)  test  was 
applied  to  their  residuals.  As  presented  in  Appendix  A,  both  FSE  rate  and  AA  rate  passed 
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the  GOF  test  at  the  alpha  level  of  0.05.  Moreover,  distribution  graphs  of  the  residuals 


appeared  to  have  an  acceptably  normal  bell  shaped  curve.  Therefore,  it  was  decided  that 
30  replications  were  adequate  to  carry  out  a  successful  experiment. 


Design  of  Experiment 

The  literature  review  identified  that  learning  curves,  reliability  growth,  and  PHM 
related  factors  had  the  potential  of  influencing  the  MOPs.  Therefore,  after  randomly 
changing  these  candidate  variables,  several  pilot  simulation  runs  were  executed  to  gain  a 
better  insight  into  their  behaviors.  Based  on  the  initial  findings,  the  following  seven 
factors  were  determined  as  critical:  Learning  curve  rate  (LCR),  reliability  growth  rate 
(RGR),  Mean  Flight  Hours  Between  False  Alarms  (MFHBFA),  Fault  Coverage  Rate 
(FCR),  Correct  Fault  Detection  Rate  (CFDR),  Fault  Isolation  Rate  (FIR),  and  Known  Fix 
Availability  Rate  (KFAR).  Next,  the  factor  count  was  decreased  to  three  by  combining 
MFHBFA,  FCR,  CFDR,  FIR,  and  KFAR  into  a  single  PHM  composite  factor  to  reduce 
the  combinatorial  growth  of  possible  experiment  treatments.  After  determining  the  most 
important  factors,  their  associated  levels  were  set  based  on  the  literature  reviews,  official 
reports,  and  expert  views  (see  Table  17). 


Table  17.  Critical  Factors  and  Their  Associated  Levels 


Factor  /  Level 

Low 

Medium 

High 

PHM 

PHM1 

PHM2 

PHM3 

MFHBFA 

170 

MFHBFA 

270 

MFHBFA 

370 

FCR 

84% 

FCR 

90% 

FCR 

95% 

CFDR 

85% 

FCDR 

90% 

FCDR 

95% 

FIR 

79% 

FIR 

85% 

FIR 

90% 

KFAR 

85% 

KFAR 

90% 

KFAR 

95% 

Reliability  Growth  Rate  (RGR) 

RGR1 

0.00 

RGR2 

0.10 

RGR3 

0.20 

Learning  Curve  Rate  (LCR) 

LCR1 

1.00 

LCR2 

0.95 

LCR3 

0.90 

57 


Note  that  when  the  RGR  and  LCR  factors  are  both  set  at  their  low  levels,  a  system 


is  represented  with  no  reliability  growth  and  no  task  learning  curve  time  reductions. 

Since  there  were  three  factors  with  three  levels,  a  full  factorial  design  was 
selected  to  examine  all  possible  combinations  of  the  factors  and  find  out  the  cause  and 
effect  relationships  between  them  and  MOPs.  As  presented  in  Table  18,  27  experiment 
runs  or  simulation  scenarios  were  obtained  as  a  result  of  the  3x3x3  full  factorial  design. 


Table  18.  Design  of  Experiment 


Run /Factor 

PHM  level 

Reliability  Growth 
Rate  (RGR) 

Learning  Curve 
Rate  (LCR) 

111 

PHM1 

RGR1 

LCR1 

112 

PHM1 

RGR1 

LCR2 

113 

PHM1 

RGR1 

LCR3 

121 

PHM1 

RGR2 

LCR1 

122 

PHM1 

RGR2 

LCR2 

123 

PHM1 

RGR2 

LCR3 

131 

PHM1 

RGR3 

LCR1 

132 

PHM1 

RGR3 

LCR2 

133 

PHM1 

RGR3 

LCR3 

211 

PHM2 

RGR1 

LCR1 

212 

PHM2 

RGR1 

LCR2 

213 

PHM2 

RGR1 

LCR3 

221 

PHM2 

RGR2 

LCR1 

222 

PHM2 

RGR2 

LCR2 

223 

PHM2 

RGR2 

LCR3 

231 

PHM2 

RGR3 

LCR1 

232 

PHM2 

RGR3 

LCR2 

233 

PHM2 

RGR3 

LCR3 

311 

PHM3 

RGR1 

LCR1 

312 

PHM3 

RGR1 

LCR2 

313 

PHM3 

RGR1 

LCR3 

321 

PHM3 

RGR2 

LCR1 

322 

PHM3 

RGR2 

LCR2 

323 

PHM3 

RGR2 

LCR3 

331 

PHM3 

RGR3 

LCR1 

332 

PHM3 

RGR3 

LCR2 

333 

PHM3 

RGR3 

LCR3 
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Output  Analysis 

Tests  of  the  AN  OVA  Assumptions 

Designing  a  3x3x3  full  factorial  DOE,  each  scenario  was  replicated  for  30  times 
and  output  data  were  transferred  into  the  JMP  for  further  statistical  analyses.  ANOVA 
was  the  main  analysis  applied  to  the  data.  At  the  beginning  of  the  ANOVA,  following 
assumptions  were  checked  to  see  whether  it  was  appropriate  to  use  ANOVA  or  not:  1) 
Normality,  2)  Constant  variance,  3)  Independence. 

The  first  assumption  check  was  the  normality  of  residuals.  After  all  experiments 
were  completed,  Shapiro-Wilk  Goodness  of  Fit  test  was  applied  to  the  residuals  of  FSE 
rate  and  AA  rate.  As  presented  in  Figure  32  and  Figure  33  in  Appendix  B,  at  the  alpha 
level  of  0.05,  neither  of  them  passed  the  test.  However,  their  distribution  histograms 
visually  appeared  normal.  Therefore,  it  was  assumed  that  residuals  of  the  MOPs  were 
approximately  normally  distributed. 

Secondly,  the  assumption  of  constant  variance  was  checked.  Scatter  plots  of  the 
FSE  rate  and  AA  rate  in  Figure  34  and  Figure  35  in  Appendix  B  demonstrated  that 
residuals  were  homogenous  throughout  the  sample  and  variability  in  the  measurement 
error  was  constant. 

Fastly,  independence  was  checked.  Residuals’  overlay  plots  in  Figure  36  and 
Figure  37  in  Appendix  B  illustrated  that  residuals  were  not  following  a  trend.  Therefore, 
it  was  decided  that  the  assumption  of  independence  was  satisfied. 
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ANOVA  Analysis  for  AA  Rate 

After  assumptions  were  tested,  ANOVA  analysis  was  conducted  to  examine  the 
effects  of  the  factors  on  the  MOPs.  Then,  the  research  questions  determined  in  Chapter  1 
were  answered. 

First,  the  AA  rate  was  analyzed.  The  ANOVA  analysis  of  the  AA  rate  in  Table  19 
shows  that  the  model  explains  nearly  99.2  percent  of  the  total  variability.  Overall  F- 
Test’s  p-value  is  smaller  than  0.05,  which  means  that  the  model  is  statistically  significant 
at  the  95%  level  in  explaining  the  variability  in  the  AA  rate. 


Table  19.  ANOVA  Results  for  the  AA  Rate 


Summary  of  Fit 

RSquare 

0.992529 

RSquare  Adj 

0.992359 

Root  Mean  Square  Error 

0.006066 

Mean  of  Response 

0.653882 

Observations  (or  Sum  Wgts) 

810 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

18 

3.8666639 

0.214815 

5837.830 

Error 

791 

0.0291064 

0.000037 

Prob  >  F 

C.  Total 

809 

3.8957703 

<.0001* 

Lack  Of  Fit 

Sum  of 

Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Lack  Of  Fit 

8 

0.00826103 

0.001033 

38.7879 

Pure  Error 

783 

0.02084540 

0.000027 

Prob  >  F 

Total  Error 

791 

0.02910643 

<.0001* 

Max  RSq 

0.9946 


The  effect  tests  in  Table  20  show  that  all  factors  and  their  associated  two-way 
interactions  have  statistically  significant  impacts  on  the  AA  rate  at  the  95%  level.  Having 
the  highest  F  Ratio,  PHM  is  the  most  influential  factor  on  the  AA  rate. 
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Table  20.  Effect  Tests  for  AA  Rate 


Effect  T  ests 

Source 

Nparm 

DF 

Sum  of 
Squares 

F  Ratio 

Prob  >  F 

PHM  level 

2 

2 

1.9823738 

26936.62 

<.0001* 

RG  rate 

2 

2 

0.2538052 

3448.721 

<.0001* 

LC  rate 

2 

2 

1.5777275 

2143826 

<.0001* 

PHM  level'RG  rate 

4 

4 

0.0062650 

42.5644 

<.0001* 

PHM  levePLC  rate 

4 

4 

0.0356717 

242.3548 

<.0001* 

RG  rate’LC  rate 

4 

4 

0.0108208 

73.5166 

<.000  r 

Following  the  effect  tests,  plots  of  the  Least  Squares  (LS)  Means  were  produced 
and  a  Tukey  test  was  conducted.  While  the  LS  Means  plot  is  a  visual  test  to  see  the 
relative  differences  in  the  response,  a  Tukey  test  provides  a  quantitative  test  serving  the 
same  purpose. 

LS  Means  plots  in  Figure  14  visually  show  each  factor’s  individual  impact  on  the 
AA  rate.  According  to  these  plots  and  Tukey  tests  in  Appendix  C,  all  factors  have 
statistically  significant  impact  on  the  AA  rate.  Moreover,  the  PHM  level  and  LC  rate 
have  greater  influence  on  the  AA  rate  than  does  RG  rate. 

]  LS  Means  Plot  a  LS  Means  Plot  a  LS  Means  Plot 

0.75 
1  0.7 

^  0,65 
S  O'6 
<  0.55 
0.5 

Figure  14.  LS  Means  Plots  of  the  Factors 

Figure  15,  Figure  16,  and  Figure  17  depict  two-way  interactions  of  the  factors  and 
their  associated  impacts  on  the  AA  rate.  Both  these  plots  and  Tukey  tests  presented  in 
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the  Appendix  C  indicate  that  two-way  interactions  have  statistically  significant 


differences  at  all  levels. 


RGRl  1  RGR2  RGR3 

RG  rate 


Figure  15.  LS  Means  Plot  of  the  PHM  -RG  Rate  Interaction 


LCR1  LCR2  LCR3 

LC  rate 


Figure  16.  LS  Means  Plot  of  the  PHM  -LC  Rate  Interaction 


LS  Means  Plot 


Figure  17.  LS  Means  Plot  of  the  RG  Rate-LC  Rate  Interaction 
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AA  rates  which  were  achieved  under  27  different  scenarios  are  charted  in 


Figure  18  labeled  with  levels  for  PHM,  RG  rate,  and  LCR  rate.  Supporting  the  ANOVA 
analysis  and  effect  tests,  Figure  18  also  demonstrates  that  all  factors  and  their  associated 
levels  significantly  affect  the  AA  rate.  When  each  factor’s  individual  impact  is  examined 
in  Figure  18,  the  PHM  and  LC  rate  seem  more  significant  than  the  RG  rate.  Additionally, 
the  lowest  AA  rate  is  realized  as  49.47%,  when  all  factors  are  at  low  values  (111  run)  and 


the  highest  AA  rate  is  realized  as  76.36%,  when  all  factors  are  at  high  levels  (333  run). 
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Figure  18.  AA  Rates  under  Different  Factor  Combinations 


ANOVA  Analysis  for  FSE  Rate 

As  presented  in  Table  21,  the  adjusted  R  Square  value  of  the  model  is  almost  97.9 
percent,  which  means  that  the  model  explains  97.9  percent  of  the  total  variability.  The 
overall  F-Test’s  p-value  is  smaller  than  0.05,  which  indicates  that  the  model  is 
statistically  significant  at  the  95%  level  in  explaining  the  variability. 
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Table  21.  ANOVA  Results  for  the  FSE  Rate 


Summary  of  Fit 

RSquare 

0.979669 

RSquare  Adj 

0.979206 

Root  Mean  Square  Error 

0.004578 

Mean  of  Response 

0.787194 

Observations  (or  Sum  Wgts) 

810 

Analysis  of  Variance 

Source 

DF 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

18 

0.79879037 

0.044377 

2117481 

Error 

791 

0.01657744 

0.000021 

Prob  >  F 

C.  Total 

809 

0.81536781 

<.0001’ 

Lack  Of  Fit 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Lack  Of  Fit 

8 

0.00385525 

0.000482 

29.6594 

Pure  Error 

783 

0.01272218 

0.000016 

Prob  >  F 

Total  Error 

791 

0.01657744 

<.0001* 

Max  RSq 

0.9844 


The  effect  tests  in  Table  22  show  that  all  factors  and  their  associated  two-way 
interactions  are  statistically  significant  on  the  FSE  rate.  Having  the  highest  F  Ratio,  the 
PHM  is  the  statistically  most  significant  factor. 


Table  22.  Effect  Tests  for  FSE  Rate 


Effect  T  ests 


Source 

Nparm 

DF 

Sum  of 
Squares 

F  Ratio 

Prob  >  1 

PHM  level 

2 

2 

0.31681864 

7558.574 

<.ooo  r 

RG  rate 

2 

2 

0.05460011 

1302.635 

<.ooor 

LC  rate 

2 

2 

0.28555239 

6812.632 

<.ooo  r 

PHM  leverRG  rate 

4 

4 

0.01612347 

192.3347 

<.ooor 

PHM  level*LC  rate 

4 

4 

0.11056090 

1318.866 

<.ooor 

RG  rate’LC  rate 

4 

4 

0.01513487 

180.5418 

<.ooor 

The  LS  Means  plots  in  Figure  19  show  each  factor’s  individual  impact  on  the  FSE 
rate.  As  the  plots  depict,  PHM  and  EC  rate  have  greater  influence  on  the  FSE  rate  than 
does  RG  rate.  However,  their  impacts  diminish  as  their  levels  are  increased.  Tukey  tests 
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presented  in  Appendix  D  also  indicate  that  all  levels  of  every  factor  significantly  affect 


the  FSE  rate. 


LS  Means  Plot 
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Figure  19.  LS  Means  Plots  of  the  Factors 


Figure  20  represents  the  LS  Means  plot  of  PHM-RG  rate  interactions.  According 


to  the  plot,  when  the  PHM  level  is  PHM3,  the  impact  of  RG  rate  on  the  FSE  rate  is 


minimal.  Furthermore,  Tukey  tests  in  the  Appendix  D  indicate  that  there  is  no  statistically 


significant  difference  between  (PHM2-RG3)  and  (PHM3-RGR1). 

LS  Means  Plot 

PHM1-*- 
PHM2-*- 
PHM3 


RGR1  RGR2  RGR3 

RG  rate 


Figure  20.  LS  Means  Plot  of  the  PHM  Level-RG  Rate  Interaction 


The  LS  Means  plot  of  the  PHM  level-LC  rate  interactions  in  Figure  21  shows  that 


when  the  PHM  level  is  increased,  the  LC  rate  affects  the  FSE  rate  at  a  diminishing  rate. 


Moreover,  at  the  PHM3  level,  increasing  the  LC  rate  from  LCR2  to  LCR3  does  not 


change  the  FSE  rate  significantly.  Supporting  this,  a  Tukey  test  in  Appendix  D  shows  that 
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there  is  no  statistically  significant  difference  between  the  following  factor  interactions: 


(PHM2-LCR3),  (PHM3-LCR3),  (PHM3-LCR2),  (PHM3-LCR1)  and  (PHM1-LCR3). 


LC  rate 


Figure  21.  LS  Means  Plot  of  the  PHM  Level-LC  Rate  Interaction 


The  LS  Means  plot  of  the  LC  rate-RG  rate  interactions  in  Figure  22  demonstrates 


that  when  the  RG  rate  is  increased,  the  LC  rate  effect  on  the  FSE  rate  diminishes  slightly. 


Similarly,  when  the  LC  rate  is  increased,  the  effect  of  the  RG  rate  on  the  FSE  diminishes 


slightly.  According  to  the  Tukey  test  in  Appendix  D,  there  is  no  statistically  significant 


difference  between  the  (RGR1-LCR3)  and  (RGR3-LCR2)  interactions. 


LC  rate 


Figure  22.  LS  Means  Plot  of  the  RG  Rate-LC  Rate  Interaction 


The  FSE  rates  achieved  under  the  27  different  scenarios  are  charted  in  Figure  23 
labeled  with  levels  for  PHM,  RG  rate,  and  LC  rate.  Parallel  to  the  ANOVA  analysis  and 
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effect  tests,  this  figure  also  shows  that  all  factors  and  their  associated  levels  significantly 
affect  the  FSE  rate.  The  lowest  FSE  rate  is  realized  as  69.28%,  when  all  factors  are  at  low 
levels  (111  run).  As  for  the  highest  FSE  rate,  it  is  realized  as  81.41%  in  the  233  run,  when 
the  PHM  level  is  medium  and  other  factors  are  high.  This  result  is  not  surprising,  since  it 
supports  Figures  21  and  22  which  demonstrate  that  at  PHM3,  high  levels  of  LC  rate  and 
RG  rate  do  not  significantly  affect  the  FSE  rate. 
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Figure  23.  FSE  Rates  under  Different  Factor  Combinations 


Analysis  of  the  PHM  Level 

As  stated  in  the  DOE  section,  the  PHM  composite  factor  is  a  combination  of  five 
sub- variables:  Mean  Flight  Hours  Between  False  Alarms  (MFHBFA),  Fault  Coverage 
Rate  (FCR),  Correct  Fault  Detection  Rate  (CFDR),  Fault  Isolation  Rate  (FIR),  and 
Known  Fix  Availability  Rate  (KFAR).  The  results  of  ANOVA  analysis  indicated  that  the 
PHM  was  the  most  significant  factor  on  both  FSE  rate  and  AA  rate.  Therefore,  some 
further  analysis  was  conducted  to  investigate  the  individual  impact  of  its  sub-variables  on 
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the  sortie  generation  process.  For  this  purpose,  RG  rate  and  LC  rate  were  set  to  their 
initial  values  and  some  one-factor  scenarios  were  examined. 

First,  effects  of  false  alarms  on  the  total  unscheduled  maintenance  time  were 
investigated  by  assigning  some  arbitrary  values  to  MFHBFA  global  variable  in  the 
simulation  model  while  all  other  factors  were  kept  constant.  As  shown  in  Figure  24,  when 
the  MFHBFA  is  increased,  total  time  spent  for  unscheduled  maintenance  decreases  since 
unnecessary  maintenance  is  avoided.  The  decrease  in  the  maintenance  time  occurs  at  a 
diminishing  rate.  After  the  MFHBFA  reaches  70  hours,  the  improvement  in  the 
maintenance  time  dramatically  slows  down;  after  370  hours,  it  almost  stops,  which  means 
the  false  alarms  number  is  very  low  and  they  are  not  significant  in  the  model.  Like  total 
maintenance  time,  FSE  rate  and  AA  rate  improve  diminishingly,  as  MFFBFA  is 
increased  gradually  over  time  (See  Figure  25). 


Figure  24.  Total  Maintenance  Time  vs.  MFHBFA 
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Figure  25.  Impact  of  MFHBFA  on  the  FSE  Rate  and  AA  Rate 


ANOVA  results  in  Appendix  E  also  support  the  findings  in  Figure  24  and  25. 
They  show  that  increasing  the  MFHBFA  from  170  hours  to  370  hours  provides  only 
small  statistically  significant  improvements  in  the  maintenance  time,  AA  rate,  and  FSE 
rate. 

Secondly,  individual  impacts  of  the  FCR,  CFDR,  FIR  and  KFAR  were  examined. 
To  see  their  impact  on  the  MOPs,  they  were  individually  set  to  three  different  levels  and 
the  simulation  model  was  replicated  30  times.  Figures  26,  27,  28,  and  29  illustrate  their 
impacts  on  the  FSE  rate  and  AA  rate,  and  Appendices  F  to  I  include  their  associated 
ANOVA  analyses.  Both  the  figures  and  ANOVA  results  indicate  that  all  sub-variables 
have  statistically  significant  impacts  on  the  MOPs.  Furthermore,  the  KFAR  is  the  most 
significant  one  among  them.  When  the  KFAR  is  increased  from  85%  to  95%,  AA  rate 
improves  from  49.47%  to  58.7%  and  FSE  rate  improves  from  69.28%  to  76.19%.  This 
finding  supports  the  importance  of  the  completeness  of  the  F-35’s  technical  documents. 
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■  FSErate  BAA  rate 
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Figure  26.  Impact  of  FCR  on  the  FSE  Rate  and  AA  Rate 


Figure  27.  Impact  of  CFDR  on  the  FSE  Rate  and  AA  Rate 
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Figure  28.  Impact  of  FIR  on  the  FSE  Rate  and  AA  Rate 


■  FSE  rate  BAA  rate 


KFAR 


Figure  29.  Impact  of  KFAR  on  the  FSE  Rate  and  AA  Rate 


Conclusion 

This  chapter  started  with  a  definition  of  the  key  measures  of  performance  (MOP) 
and  continued  with  the  development  of  a  designed  experiment.  The  DOE  was  built  as  a 
3x3x3  full  factorial  design  and  all  possible  combinations  of  the  three  factors  were 
replicated  for  30  times  through  the  simulation  model.  Then,  ANOVA  analyses  were 
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conducted  to  investigate  the  effects  of  the  factors  on  the  MOPs.  Additionally,  some 
further  analyses  were  conducted  to  examine  the  individual  impacts  of  the  PHM’s  sub¬ 
variables.  Therefore,  it  was  demonstrated  that  all  factors  were  significantly  influential  on 
the  MOPs.  The  next  chapter  summarizes  the  research  and  gives  recommendations  for 
further  studies. 
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V.  Conclusions  and  Recommendations 


Introduction 

Previous  chapters  described  the  research  topic,  provided  related  literature  reviews, 
explained  the  simulation  model,  defined  the  MOPs  and  critical  factors,  and  analyzed  the 
model  outputs.  This  chapter  summarizes  the  overall  research,  explains  the  results  and 
gives  recommendations  for  future  research. 

Research  Summary 

This  research  investigated  the  sortie  generation  process  of  sixteen  F-35  aircraft 
under  an  autonomic  logistics  system  at  a  notional  base  in  Turkey.  For  this  purpose,  a 
discrete  event  simulation  model  of  the  sortie  generation  process  was  built  in  Arena® 
software.  The  key  parts  of  the  model  were  developed  in  view  of  the  most  recent  practices 
regarding  the  F-35  and  its  logistics  system. 

Based  on  the  literature  researches  and  expert  views,  the  aircraft  availability  rate 
and  flying  scheduling  effectiveness  rate  were  determined  as  the  key  measure  of 
performance  (MOP)  for  the  sortie  generation  process.  PHM  level,  reliability  growth  rate, 
and  learning  curve  rate  were  chosen  as  the  critical  factors  potentially  affecting  these 
measures.  Then,  a  3x3x3  full  factorial  experiment  was  designed  to  analyze  the  simulation 
outputs.  The  simulation  run  length  was  determined  as  five  years  (1825  days)  of  24-hour 
operations  and  each  of  the  27  scenarios  were  replicated  30  times.  Simulation  outputs 
were  imported  to  JMP®  software  and  ANOVA  analysis  was  conducted  to  see  the 
possible  cause  and  effect  relationships  between  factors  and  MOPs. 
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Research  Conclusion 


The  literature  reviews  indicated  that  the  AL  concept  has  been  a  popular  research 
subject  since  the  first  introduction  of  the  F-35.  However,  most  prior  studies  investigated 
the  AL  concept  from  a  notional  basis  due  to  a  lack  of  real  world  data. 

Initial  delivery  of  the  F-35s  will  be  made  to  Turkey  in  the  near  future.  However, 
official  reports  show  that  ALIS  is  still  far  from  achieving  the  desired  level  of  PHM 
functionality.  Presently,  the  prognostic  capability  of  the  PHM  system  is  not  functional, 
and  diagnostic  capability  is  functional  only  with  serious  malfunctions.  Keeping  these 
realities  in  mind,  this  study  aimed  to  explain  sortie  generation  process  of  the  F-35  using 
actual  data. 

For  this  purpose,  critical  factors  were  determined  and  their  potential  impacts  on 
the  sortie  generation  process  of  the  F-35  were  examined  in  term  of  AA  rate  and  FSE  rate. 
The  relationship  between  critical  factors  and  MOPs  were  analyzed  through  ANOVA. 

The  ANOVA  results  in  Chapter  4  showed  that  all  factors  and  their  possible 
interactions  had  statistically  significant  impact  on  the  AA  rate.  The  lowest  AA  rate  was 
obtained  as  49.06%  at  the  PHM1-RGR1-LCR1  treatment,  when  all  factors  were  at  their 
low  levels.  The  highest  AA  rate  was  achieved  as  76.26%  at  PHM3-RGR3-LCR3 
treatment,  when  all  factors  were  set  to  their  high  levels. 

Also,  all  factors  were  statistically  significant  on  the  FSE  rate.  However,  there 
were  not  significant  differences  between  some  of  their  interactions.  When  the  PHM  level 
was  increased  from  PHM2  to  PHM3,  changes  in  the  RG  rate  and  LC  rate  did  not 
significantly  affect  the  FSE  rate.  Parallel  to  this  finding,  the  highest  FSE  rate  was 
achieved  as  80.24%  at  PHM2-RGR3-LCR3  treatment,  when  PHM  level  was  medium, 
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and  other  factors  were  high.  The  lowest  FSE  rate  was  realized  as  67.63%  at  PHM1- 
RGR1-LCR1  treatment,  when  all  factors  were  at  their  low  values. 

Another  important  finding  of  the  research  was  on  the  PHM.  The  PHM  composite 
factor  appeared  as  the  statistically  most  significant  predictor  variable  within  the  model. 
While  four  of  the  PHM  sub-variables  had  statistically  significant  impact  on  the  MOPs  at 
all  levels,  the  MFHBFA  (Mean  Flight  Hours  Between  False  Alarms)  had  a  small 
statistically  significant  effect  on  the  MOPs.  Therefore,  it  may  be  concluded  that  the 
current  level  of  the  MFHBFA  (which  is  170  hours)  is  reasonably  good  for  critical 
failures.  The  potential  gains  from  the  improvement  of  the  MFHBFA  are  relatively 
smaller  than  the  potential  gains  from  other  sub-variables. 

Recommendations  for  Further  Study 

While  building  the  simulation  model,  some  important  assumptions  and  limitations 
were  defined.  Investigating  them  offers  potential  to  improve  this  research  in  many  ways. 

First,  some  parts  of  the  model  were  developed  using  other  aircraft’s  data  due  to 
the  lack  of  real-world  F-35  data.  Future  researches  may  attempt  to  obtain  real  F-35  data 
and  update  the  associated  parts  of  the  model  accordingly. 

Second,  this  research  only  investigated  mission  critical  failures  and  false  positives 
within  the  sortie  generation  process.  Additionally,  the  PHM  system’s  prognostic 
capability  was  not  modeled;  since  it  was  not  functional  at  the  time  this  research  was 
conducted.  Adding  other  failure  types,  false  negatives  and  prognostic  capability  into  the 
model  may  help  it  to  produce  more  realistic  outputs. 
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Third,  analyzing  the  supply  activities  was  beyond  the  scope  of  this  study. 
However,  the  supply  process  is  as  important  as  maintenance  in  generating  sorties. 
Moreover,  the  AL  system  aims  to  change  and  improve  the  supply  activities  considerably. 
Therefore,  analyzing  the  sortie  generation  process  from  a  supply  standpoint  may  provide 
benefits  for  the  decision  makers  in  the  supply  area. 

Fourth,  all  resources  were  assumed  to  have  infinite  capacity  in  the  model. 
Therefore,  queuing  did  not  occur  and  so  queuing  effects  on  the  sortie  generation  process 
were  not  analyzed.  Adding  equipment  and  human  resources  into  the  model  and 
investigating  the  impacts  of  the  resource  levels  on  the  MOPs  may  produce  beneficial 
outputs  for  the  decision  makers  in  the  equipment  and  personnel  management  areas. 

Next,  reliability  growth  and  learning  curves  were  two  important  phenomena 
embedded  into  the  model  logic.  While  the  reliability  growth  rate  was  calculated  based  on 
actual  F-35  data,  the  learning  curve  rate  was  defined  as  an  assumption.  Future  researchers 
may  focus  on  these  concepts  more  deeply  and  update  the  model  according  to  actual  data 
when  available. 

Finally,  while  conducting  this  study,  the  researcher  was  not  able  to  visit  F-35 
bases  and  review  the  overall  sortie  generation  process  in  the  field.  Also,  accessing  F-35 
field  experts  was  problematic.  Making  field  visits  and  interviewing  with  logistics 
personnel  would  make  great  contributions  to  the  research.  In  particular,  the  modelling 
part  could  be  more  closely  aligned  to  actual  operations.  We  strongly  recommend  future 
researchers  to  make  field  visits  to  the  F-35  bases,  before  building  the  simulation  model. 
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Appendix  A:  Normality  Test  Results 


-0.015  -0.01  -0.005  0  0.005  0.01  0.015  0.02 


—  Normal(2.6e-17.0.00771) 


J  Quantiles  a*  Summary  Statistics 


100.0% 

maximum 

0.0178633333 

Mean 

2.591e-17 

99.5% 

0.0178633333 

Std  Dev 

0.0077089 

97.5% 

0.0178633333 

Std  Err  Mean 

0.0014074 

90.0% 

0.0139733333 

Upper  95%  Mean 

0.0028785 

75.0% 

quartile 

0.0040383333 

Lower  95%  Mean 

-0,002879 

50.0% 

median 

-0.001236667 

N 

30 

25.0% 

quartile 

-0.003611667 

10.0% 

-0.011556667 

2.5% 

-0.014136667 

0.5% 

-0.014136667 

0.0% 

minimum 

-0.014136667 

A  ▼  Fitted  Normal 
A  Parameter  Estimates 


Type 

Parameter 

Estimate 

Lower  95%  Upper  95% 

Location 

P 

2.591e-17 

-0.002879 

0.0028785 

Dispersion 

0 

0.0077089 

0.0061394 

0.0103631 

-2log(Likelihood)  =-207.786776792714 

J  Goodness-of-FitTest 

Shapiro- Wilk  WTest 

W  Prob<W 

0.960317  0.3156 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p- 
values  reject  Ho. 


Figure  30.  Shapiro- Wilk  Test  Results  for  the  Residuals  of  FSE  Rate 


—  Normal(-7e-18. 0.00835) 


Quantiles 

▼  Summary  Statistics 

100.0% 

maximum 

0.0201933333 

Mean 

-7.4e-18 

99.5% 

0.0201933333 

Std  Dev 

0.0083524 

97.5% 

0.0201933333 

Std  Err  Mean 

0.0015249 

90.0% 

0.0164933333 

Upper  95%  Mean 

0.0031188 

75.0% 

quartile 

0.0031183333 

Lower  95%  Mean 

-0.003119 

50.0% 

median 

-0.000356667 

N 

30 

25.0% 

quartile 

-0.004731667 

10.0% 

-0.010566667 

2.5% 

-0.014306667 

0.5% 

-0.014306667 

0.0% 

minimum 

-0.014306667 

^  -  Fitted  Normal 
A  Parameter  Estimates 

Type  Parameter  Estimate  Lower  95%  Upper  95% 

Location  p  -7.4e-18  -0.003119  0.0031188 

Dispersion  a  0.0083524  0.0066519  0.0112282 

-2log(Likelihood)  =  -202.976311393385 

•d  G  ood  ness  -  of  -  F  i  t  T  est 

Shapiro- Wilk  WTest 

W  Prob<W 

0,930742  0.0514 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p- 
values  reject  Ho. 


Figure  31.  Shapiro- Wilk  Test  Results  for  the  Residuals  of  AA  Rate 
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Appendix  B:  Tests  of  the  ANOVA  Assumptions 


i  ▼  Residual  FSE  rate 


••  1 - 1  1  i - H*  •  • 

-J\ 

/ 

N 

\ 

_ 

0.015  -0.01  -0.005  0  0.005  0.01  0.015  0.02 


- Normalf3.2e-17. 0.00453) 


Quantiles 

A  ▼  Summary  Statistics 

100.0% 

maximum 

0.0193553086 

Mean 

3.248e-17 

99.5% 

0.0132588086 

StdDev 

0.0045267 

97.5% 

0.0089740309 

Std  Err  Mean 

0.0001591 

90.0% 

0.0057259012 

Upper  95%  Mean 

0.0003122 

75.0% 

quartile 

0.0029409568 

Lower  95%  Mean 

-0.000312 

50.0% 

median 

-0.000199321 

N 

810 

25.0% 

quartile 

-0.003019969 

10.0% 

-0.005882173 

2.5% 

-0.008650701 

0.5% 

-0.011433145 

0.0% 

minimum 

-0.014212099 

Fitted  Normal 


A  Parameter  Estimates 

Type 

Parameter 

Estimate 

Lower  95% 

Upper  95% 

Location 

P 

3.248el7 

-0.000312 

0.0003122 

Dispersion 

a 

0.0045267 

0.0043165 

0.0047586 

-2log(Likelihood)  =  -6446.68405759912 

4)  Goodness-of-Fit  T est 

Shapiro- Wilk  W  Test 

W  Prob<W 

0.995821  0.0282* 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p- 
values  reject  Ho. 


Figure  32.  Shapiro-Wilk  Test  Results  for  FSE  Rate  (Normality) 


-  Residual  AA  rate 


4]  Quantiles  4]  ▼  Summary  Statistics 


100.0% 

maximum 

0.0206918519 

Mean 

-3.13e-17 

99.5% 

0.01627435 

StdDev 

0.0059982 

97.5% 

0.0125498241 

Std  Err  Mean 

0  0002108 

90.0% 

0.0077116667 

Upper  95%  Mean 

0.0004137 

75.0% 

quartile 

0.0038193519 

Lower  95%  Mean 

-0.000414 

50.0% 

median 

-0.00005963 

N 

810 

25.0% 

quartile 

-0.00  3  722778 

10.0% 

-0.007486 

2.5% 

-0.012189509 

0.5% 

-0.017574719 

0.0% 

minimum 

-0.022363333 

4]  ▼  Fitted  Normal 


A  Parameter  Estimates 

Type 

Parameter 

Estimate 

Lower  95% 

Upper  95% 

Location 

P 

-3.13e-17 

-0.000414 

0.0004137 

Dispersion 

o 

0.0059982 

0.0057196 

0.0063055 

-2log(Likelihood)  =  -5990.72162007459 

a  Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 

0.996718  0.0942 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p- 


- Normal(-3e-17, 0.006) 


values  reject  Ho. 


Figure  33.  Shapiro-Wilk  Test  Results  for  AA  rate  (Normality) 


Residual  by  Predicted  Plot 


Inner 

range 


0.66  0.68  0.7  0.72  0.74  0.76  0.78  0.8  0.82 

FSE  rate  Predicted 

Figure  34.  Residual  Plot  of  the  FSE  Rate  (Constant  Variance) 


Extreme 

range 


0.015 


0.010 


0.005 


0.000  -  - 


-0.005 


-0.010 


-0.015 
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Mrate  Residual 


Residual  by  Predicted  Plot 


0.020  ■ 

0.015  * 

X 

• 

I 

I 

I 

-0.015  I 

-0.020  ■ 

0.5  0.55  0.6  0.65  0.7  0.75 

AA  rate  Predicted 


0.010 

0.005 

0.000 

-0.005 

-0.010 


Extreme 

range 


Inner 

range 


Figure  35.  Residual  Plot  of  the  A  A  Rate  (Constant  Variance) 
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Rows 


Figure  36.  Overlay  Plot  of  the  FSE  Rate’s  Residuals  (Independence) 


0.02 

0.015 

0.01 

0.005 
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-0.005 

-0.01 

-0.015 
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300  400 

Rows 
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Figure  37.  Overlay  Plot  of  the  AA  Rate’s  Residuals  (Independence) 
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Appendix  C:  Tukey  Test  Results  for  A  A  Rate 


Table  23.  Tukey  Tests  for  PHM  Level,  RG  Rate,  and  LC  Rate 


-■  LSMeans  Differences  Tukey  HSD 

a=  0.050  Q=  2.34813 

LSMean[j] 


A  ▼  LSMeans  Differences  Tukey  HSD 

a=  0.050  Q=  2.34813 

LSMean[j] 


A  ▼  LSMeans  Differences  Tukey  HSD 

a=  0.050  Q=  2.34813 

LSMean  [fl 


Mean[i]-Mean[j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

PHMl 

PHM2 

PHM3 

Mean[i]-Mean(j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

RGR1 

RGR2 

RGR3 

Mean[i]-Mean[j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

LCRl 

LCR2 

LCR3 

PHMl 

0 

-0.0635 

-0.1211 

RGR1 

0 

-0.0229 

-0.0433 

LCRl 

0 

-0.0578 

-0.108 

0 

0.00052 

0.00052 

0 

0.00052 

0.00052 

0 

0.00052 

0.00052 

0 

-0.0647 

-0.1224 

r= 

0 

-0.0241 

-0.0446 

j- 

0 

-0.059 

-0.1092 

c 

? 

0 

-0.0622 

-0.1199 

c 

s 

0 

-0.0216 

-0.0421 

c 

s 

0 

-0.0566 

-0.1068 

2 

PHM2 

0.06347 

0 

-0.0577 

2 

RGR2 

0.02286 

0 

-0.0205 

2 

LCR2 

0.05781 

0 

-0.0502 

2 

0.00052 

0 

0.00052 

0.00052 

0 

0.00052 

0.00052 

0 

0.00052 

0.06224 

0 

-0.0589 

0.02164 

0 

-0.0217 

0.05658 

0 

-0.0514 

0.06469 

0 

-0.0564 

0.02409 

0 

-0.0192 

0.05904 

0 

-0.049 

PHM3 

0.12113 

0.05766 

0 

RGR3 

0.04334 

0.02047 

0 

LCR3 

0.10802 

0.05021 

0 

0.00052 

0.00052 

0 

0.00052 

0.00052 

0 

0.00052 

0.00052 

0 

0.11991 

0.05644 

0 

0.04211 

0.01925 

0 

0.10679 

0.04898 

0 

0.12236 

0.05889 

0 

0.04456 

0.0217 

0 

0.10924 

0.05143 

0 

Least 

Least 

Least 

Level 

Sq  Mean 

Level 

Sq  Mean 

Level 

Sq  Mean 

PHM3  A 

0.71348037 

RGR3 

A  0.67515185 

LCR3  A 

0.70662296 

PHM2 

B  0.65581704 

RGR2 

B  0.65467926 

LCR2 

B  0.65641630 

PHMl 

C  0.59234815 

RGR1 

C  0.63181444 

LCRl 

C  0.59860630 

Levels  not 

connected  by  same  letter  are  significantly  different. 

Levels 

not  connected  by  same  letter  are  significantly  different. 

Levels  not 

connected  by  same 
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Table  24.  Tukey  Test  for  Two-way  Interactions  of  PHM  Level  and  RG  Rate 


t  LSMeans  Differences  Tukey  HSD 

a=  0.050  Q=  3.11039 


LSMean[j] 


Mean[i]-Mean[j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

PH  MIR 
GR1 

PH  MI  R 
GR2 

PH  MI  R 
GR3 

PHM2.R 

GR1 

PHM2R 

GR2 

PHM2R 

GR3 

PHM3.R 

GR1 

PHM3.R 

GR2 

PHM3.R 

GR3 

PHM1.RGR1 

0 

-0.0272 

-0.0514 

-0.0678 

-0.0897 

-0.1115 

-0.1292 

-0.1487 

-0.1641 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

-0.03 

-0.0542 

-0.0706 

-0.0925 

-0.1143 

-0.132 

-0.1515 

-0.1669 

0 

-0.0244 

-0.0486 

-0.0649 

-0.0869 

-0.1087 

-0.1264 

-0.1459 

-0.1613 

PHM1  RGR2 

0.02718 

0 

-0.0242 

-0.0406 

-0.0626 

-0.0843 

-0.102 

-0.1215 

-0.1369 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.02436 

0 

-0.027 

-0.0434 

-0.0654 

-0.0871 

-0.1048 

-0.1243 

-0.1397 

0  02999 

0 

-0.0214 

-0.0378 

-0.0597 

-0  0815 

-0.0992 

-0.1187 

-0  1341 

PHM1  RGR3 

0.0514 

0.02422 

0 

-0.0164 

-0.0383 

-0.0601 

-0.0778 

-0.0973 

-0.1127 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.04859 

0.02141 

0 

-0.0192 

-0.0411 

-0.0629 

-0.0806 

-0.1001 

-0.1155 

0.05421 

0.02704 

0 

-0.0135 

-0.0355 

-0.0573 

-0.075 

-0.0945 

-0.1099 

PHM2RGR1 

0.06776 

0.04059 

0.01636 

0 

-0.022 

-0.0437 

-0.0615 

-0.0809 

-0.0963 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.06495 

0.03777 

0.01355 

0 

-0.0248 

-0.0465 

-0.0643 

-0.0837 

-0.0991 

0.07057 

0.0434 

0.01917 

0 

-0.0192 

-0.0409 

-0.0586 

-0.0781 

-0.0935 

PHM2RGR2 

0.08973 

0.06255 

0.03833 

0.02197 

0 

-0.0218 

-0.0395 

-0.0589 

-0.0744 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.08692 

0.05974 

0.03552 

0.01915 

0 

-0.0246 

-0.0423 

-0.0617 

-0.0772 

0.09254 

0.06536 

0.04114 

0.02478 

0 

-0.019 

-0.0367 

-0.0561 

-0.0716 

PHM2RGR3 

0.11149 

0.08432 

0.06009 

0.04373 

0.02176 

0 

-0.0177 

-0.0372 

-0.0526 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.10868 

0.0815 

0.05728 

0.04092 

0.01895 

0 

-0.0205 

-0.04 

-0.0554 

0.1143 

0.08713 

0.06291 

0.04654 

0.02458 

0 

-0.0149 

-0.0344 

-0.0498 

PHM3  RGR1 

0.12921 

0.10204 

0.07781 

0.06145 

0.03948 

0.01772 

0 

-0.0195 

-0.0349 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.1264 

0.09922 

0.075 

0.05864 

0.03667 

0.01491 

0 

-0.0223 

-0.0377 

0.13202 

0.10485 

0.08063 

0.06426 

0.0423 

0.02053 

0 

-0.0166 

-0.0321 

PHM3  RGR2 

0.14866 

0.12149 

0.09727 

0.0809 

0.05894 

0.03717 

0.01945 

0 

-0.0154 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.14585 

0.11868 

0.09445 

0.07809 

0.05612 

0.03436 

0.01664 

0 

-0.0182 

0.15148 

0.1243 

0.10008 

0.08372 

0.06175 

0.03998 

0.02226 

0 

-0.0126 

PHM3  RGR3 

0.16409 

0.13692 

0.1127 

0.09633 

0.07437 

0.0526 

0.03488 

0.01543 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.16128 

0.13411 

0.10988 

0.09352 

0.07155 

0.04979 

0.03207 

0.01262 

0 

0.16691 

0.13973 

0.11551 

0.09915 

0.07718 

0.05541 

0.03769 

0.01824 

0 

Least 

Level  Sq  Mean 


PHM3.RGR3  A 
PHM3.RGR2  B 
PHM3.RGR1  C 
PHM2.RGR3  D 

PHM2RGR2 
PHM2.RGR1 
PHM1.RGR3 
PHM1  RGR2 
PHM1RGR1 


0.73025111 
0.71482111 
0.69536889 
0.67764889 
0.6  5  5  88444 
0.63391778 
G  0.61755556 
H  0.59333222 
I  0.56615667 


Levels  not  connected  by  same  letter  are  significantly  different. 
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Table  25.  Tukey  Test  for  Two-way  Interactions  of  PHM  Level  and  LC  Rate 


i  LSMeans  Differences  Tukey  HSD 

ot=  0.050  Q=  3.11039 


LSMean[j] 


Mean[i]-Mean(j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

PHM1,L 

CR1 

PHM1L 

CR2 

PH  MI  L 
CR3 

PHM2,L 

CR1 

PHM2..L 

CR2 

PHM2.L 

CR3 

PHM3,L 

CR1 

PHM3.L 

CR2 

PHM3.L 

CR3 

PHM1  LCR1 

0 

-0.0592 

-0.1235 

-0.0661 

-0.1275 

-0.1795 

-0.1353 

-0.1882 

-0.2226 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

-0.062 

-0.1263 

-0.0689 

-0.1303 

-0.1823 

-0.1381 

-0.191 

-0.2254 

0 

-0.0564 

-0.1206 

-0.0633 

-0.1246 

-0.1767 

-0.1325 

-0.1854 

-0.2197 

PHM1  LCR2 

0.05921 

0 

-0.0642 

-2  0069 

-0.0682 

-0.1203 

-0.0761 

-0.129 

-0.1633 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0564 

0 

-0.0671 

-0.0097 

-0.0711 

-0.1231 

-0.0789 

-0.1318 

-0.1662 

0.06202 

0 

-0.0614 

-0.0041 

-0.0654 

-0.1175 

-0.0733 

-0.1262 

-0.1605 

PHM1  LCR3 

0.12346 

0.06425 

0 

0.05733 

-0.004 

-0.056 

-0.0119 

-0.0647 

-0.0991 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.12064 

0.06143 

0 

0.05452 

-0.0068 

-0.0588 

-0.0147 

-0.0676 

-0.1019 

0.12627 

0.06706 

0 

0.06014 

-0.0012 

-0.0532 

-0.009 

-0.0619 

-0.0963 

PHM2.LCR1 

0.06613 

0.00692 

-0.0573 

0 

-0.0613 

-0.1134 

-0  0692 

-0.1221 

-0.1564 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.06332 

0.00411 

-0.0601 

0 

-0.0641 

-0.1162 

-0.072 

-0.1249 

-0.1592 

0.06894 

0.00973 

-0.0545 

0 

-0.0585 

-0.1105 

-0.0664 

-0.1193 

-0.1536 

PHM2.LCR2 

0.12746 

0.0  6825 

0.004 

0.06133 

0 

-0.052 

-0.0079 

-0.0607 

-0.0951 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.12465 

0.06544 

0.00119 

0.05852 

0 

-0.0548 

-0.0107 

-0.0636 

-0.0979 

0.13027 

0.07106 

0.00682 

0.06414 

0 

-0.0492 

-0.005 

-0.0579 

-0.0923 

PHM2.LCR3 

0.17948 

0.12027 

0.05603 

0.11336 

0.05202 

0 

j  24417 

-0.0087 

-0  0431 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.17667 

0.11746 

0.05322 

0.11054 

0.04921 

0 

0.04136 

-0.0115 

-0.0459 

0.1823 

0.12309 

0.05884 

0  11617 

0.05484 

0 

0.04698 

-0.0059 

-0.0403 

PHM3.LCR1 

0.13531 

0.0761 

0.01186 

0.06918 

0.00785 

-0  0442 

0 

-0.0529 

-j  2672 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.1325 

0.07329 

0.00904 

0.06637 

0.00504 

-0.047 

0 

-0.0557 

-0.0901 

0.13813 

0.07891 

0.01467 

0.072 

0.01066 

-0.0414 

0 

-0.0501 

-0.0844 

PHM3.LCR2 

0.1882 

0.12899 

2  06474 

0.12207 

0.06074 

2  22672 

0.05289 

0 

-0  2244 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.18539 

0.12618 

0.06193 

0.11926 

0.05793 

0.0059 

0.05007 

0 

-0.0372 

0.19101 

0.1318 

0.06756 

0.12488 

0.06355 

0.01153 

0.0557 

0 

-0.0315 

PHM3.LCR3 

0.22255 

0.16334 

0.09909 

0.15642 

0.09509 

0. 743:7 

0.08724 

0  03435 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.21974 

0.16053 

0.09628 

0.15361 

0.09228 

0.04025 

0.08443 

0.03154 

0 

0.22536 

0.16615 

0.10191 

0.15923 

0.0979 

0.04588 

0.09005 

0.03716 

0 

Least 


Level 

PHM3.  LCR3  A 
PHM3..LCR2  B 
PHM2..LCR3  C 
PHM3..LCR1  D 

PHM2..LCR2  E 

PHM1LCR3 
PHM2.LCR1 
PHM1.LCR2 
PHM1LCR1 


Sq  Mean 

0.75401000 
0.71965889 
0.71094333 
0.66677222 
0.65892000 
0.65491556 
G  0.59758778 
H  0.59067000 
I  0.53145889 


Levels  not  connected  by  same  letter  are  significantly  different. 
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Table  26.  Tukey  Test  for  Two-way  Interactions  of  RG  Rate  and  LC  Rate 


"  LSMeans  Differences  Tukey  HSD 

oc=  0.050  Q=  3.11039 


LSMeanp] 


Mean[i]-Meanp 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

RGR1  L 
CR1 

RGR1  L 
CR2 

RGR1  L 
CR3 

RGR2  L 
CR1 

RGR2  L 
CR2 

RGR2.L 

CR3 

RGR3  L 
CR1 

RGR3  L 
CR2 

RGR3 1 
CR3 

RGR1.LCR1 

0 

-0.0662 

-0.1183 

-0.0295 

-0.0856 

-0.1379 

-0.0553 

-0.1065 

-0.1526 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

-0.069 

-0.1211 

-0.0324 

-0.0884 

-0.1408 

-0.0581 

-0.1093 

-0.1555 

0 

-0.0633 

-0.1155 

-0.0267 

-0.0828 

-0.1351 

-0.0525 

-0.1037 

-0.1498 

RGR1.LCR2 

0.06616 

0 

-0.0521 

0.03662 

-0.0194 

-0.0718 

0.01087 

-0.0404 

-0.0865 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.06335 

0 

-0.0549 

0.03381 

-0.0222 

-0.0746 

0.00806 

-0.0432 

-0.0893 

0.06897 

0 

-0.0493 

0.03943 

-0.0166 

-0.069 

0.01368 

-0.0376 

-0.0837 

RGR1LCR3 

0.11829 

0.05213 

0 

0.08875 

0.03273 

-0.0197 

0.063 

0.01176 

-0.0344 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.11548 

0.04932 

0 

0.08594 

0.02991 

-0.0225 

0.06019 

0.00895 

-0.0372 

0.1211 

0.05494 

0 

0.09157 

0.03554 

-0.0168 

0.06581 

0.01457 

-0.0315 

RGR2LCR1 

0.02954 

-0.0366 

-0.0888 

0 

-0.056 

-0.1084 

-0.0258 

-0.077 

-0.1231 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.02673 

-0.0394 

-0.0916 

0 

-0.0588 

-0.1112 

-0.0286 

-0.0798 

-0.1259 

0.03235 

-0.0338 

-0.0859 

0 

-0.0532 

-0.1056 

-0.0229 

-0.0742 

-0.1203 

RGR2.LCR2 

0.08557 

0.0194 

-0.0327 

0.05603 

0 

-0.0524 

0.03028 

-0.021 

-0.0671 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.08275 

0.01659 

-0.0355 

0.05321 

0 

-0.0552 

0.02746 

-0.0238 

-0.0699 

0.08838 

0.02222 

-0.0299 

0.05884 

0 

-0.0496 

0.03309 

-0.0182 

-0.0643 

RGR2LCR3 

0.13794 

0.07178 

0.01965 

0.1084 

0.05238 

0 

0.08265 

0.03141 

-0.0147 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.0009 

0.13513 

0.06897 

0.01684 

0.10559 

0.04957 

0 

0.07984 

0.0286 

-0.0175 

0.14076 

0.07459 

0.02246 

0.11122 

0.05519 

0 

0.08547 

0.03422 

-0.0119 

RGR3.LCR1 

0.05529 

-0.0109 

-0.063 

0.02575 

-0.0303 

-0.0827 

0 

-0.0512 

-0.0974 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.0009 

0.05248 

-0.0137 

-0.0658 

0.02294 

-0.0331 

-0.0855 

0 

-0.0541 

-0.1002 

0.0581 

-0.0081 

-0.0602 

0.02856 

-0.0275 

-0.0798 

0 

-0.0484 

-0.0945 

RGR3LCR2 

0.10653 

0.04037 

-0.0118 

0.07699 

0.02097 

-j  0314 

0.05124 

0 

-0.0461 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.0009 

0.10372 

0.03756 

-0.0146 

0.07418 

0.01815 

-0.0342 

0.04843 

0 

-0.0489 

0.10934 

0.04318 

-0.0089 

0.07981 

0.02378 

-0.0286 

0.05405 

0 

-0.0433 

RGR3.LCR3 

0.15264 

0.08648 

0.03435 

0.1231 

0.06708 

0.0147 

0.09735 

0.04611 

0 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0.0009 

0 

0.14983 

0.08367 

0.03154 

0.12029 

0.06427 

0.01189 

0.09454 

0.0433 

0 

0.15546 

0.08929 

0.03716 

0.12592 

0.06989 

0.01751 

0.10017 

0.04892 

0 

Level 

Least 
Sq  Mean 

RGR3.  LCR3  A 

0.72297333 

RGR2.LCR3  B 

0.70827333 

RGR1LCR3  C 

0.68862222 

RGR3.LCR2  D 

0.67686222 

RGR2..LCR2  E 

0.65589556 

RGR1.LCR2  F 

0.63649111 

RGR3.LCR1  G 

0.62562000 

RGR2.LCR1 

H  0.59986889 

RGR1.LCR1 

I  0.57033000 

Levels  not  connected  by  same 

letter  are  siqnificai 
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Appendix  D:  Tukey  Test  Results  for  FSE  Rate 


Table  27.  Tukey  Tests  for  PHM  Level,  RG  Rate,  and  LC  Rate 


~  LSMeans  Differences  Tukey  HSD 

a=  0.050  Q=  2.34813 

LSMean[j] 


A  ▼  LSMeans  Differences  Tukey  HSD 

a=  0.050  Q=  2.34813 

LSMean[j] 


A  ▼  LSMeans  Differences  Tukey  HSD 

a=  0.050  Q=  2.34813 

LSMean[j] 


Mean[i]-Mean[j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

PHM1 

PHM2 

PHM3 

Mean[i]-Mean(j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

RGRl 

RGR2 

RGR3 

Mean[i]-Mean[j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

LCRl 

LCR2 

LCR3 

PHM1 

0 

-0.0345 

-0.0467 

RGRl 

0 

-0.0118 

-0.02 

LCRl 

0 

-0.0299 

-0.0452 

0 

0.00039 

0.00039 

0 

0.00039 

0.00039 

0 

0.00039 

0.00039 

!= 

0 

-0.0355 

-0.0476 

0 

-0.0128 

-0.0209 

r= 

0 

-0.0308 

-0.0461 

c 

* 

0 

-0.0336 

-0.0458 

c 

g 

0 

-0.0109 

-0.0191 

c 

s 

0 

-0.029 

-0.0443 

2 

PHM2 

0.03453 

0 

-0.0122 

2 

RGR2 

0.01184 

0 

-0.0082 

2 

LCR2 

0.02988 

0 

-0.0153 

0.00039 

0 

0.00039 

2 

0.00039 

0 

0.00039 

2 

0.00039 

0 

0.00039 

0.0336 

0 

-0.0131 

0.01092 

0 

-0.0091 

0.02895 

0 

-0.0163 

0.03545 

0 

-0.0112 

0.01277 

0 

-0.0072 

0.0308 

0 

-0.0144 

PHM3 

0.04669 

0.01216 

0 

RGR3 

0.02 

0.00815 

0 

LCR3 

0.04522 

0.01534 

0 

0.00039 

0.00039 

0 

0.00039 

0.00039 

0 

0.00039 

0.00039 

0 

0.04577 

0.01124 

0 

0.01907 

0.00723 

0 

0.04429 

0.01442 

0 

0.04762 

0.01309 

0 

0.02092 

0.00908 

0 

0.04614 

0.01627 

0 

Least 

Least 

Least 

Level 

Sq  Mean 

Level 

Sq  Mean 

Level 

Sq  Mean 

PHM3  A 

0.80681148 

RGR3 

A  0.79657778 

LCR3 

A  0.80738148 

PHM2 

B  0.79464889 

RGR2 

B  0.78842296 

LCR2 

B  0.79203741 

PHM1 

C  0.76012037 

RGRl 

C  0.77658000 

LCRl 

C  0.76216185 

Levels  not 

connected  by  same  letter  are  significantly  different. 

Levels 

not  connected  by  same  letter  are  significantly  different. 

Levels 

not  connected  by  same 
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Table  28.  Tukey  Test  for  Two-way  Interactions  of  PHM  Level  and  RG  Rate 


▼  LSMeans  Differences  Tukey  HSD 

ct=  0.050  Q=  3.11039 


LSMean[j] 


Mean[i]-Mean[j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

PH  MI  R 
GR1 

PH  MI  R 
GR2 

PH  M1.R 
GR3 

PHM2.R 

GR1 

PHM2.R 

GR2 

PHM2.R 

GR3 

PHM3.R 

GR1 

PHM3R 

GR2 

PHM3.R 

GR3 

PHM1  RGR1 

0 

-0.02 

-0.0332 

-0.0417 

-0.0532 

-0.0618 

-0.0609 

-0.0649 

-0.0675 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

-0.0221 

-0.0353 

-0.0438 

-0.0554 

-0.064 

-0.063 

-0.067 

-0.0696 

0 

-0.0178 

-0.0311 

-0.0395 

-0.0511 

-0.0597 

-0.0587 

-0.0627 

-0.0654 

PHM1..RGR2 

0.01995 

0 

-0.0132 

-0.0217 

-0.0333 

-0.0419 

-0.0409 

-0.0449 

-0.0475 

0.00068 

0 

0.00068 

0.00068 

0.00068 

j  j0j5S 

0.00068 

0.00068 

0.00068 

0.01783 

0 

-0.0154 

-0.0238 

-0.0354 

-0.044 

-0.043 

-0.047 

-0.0497 

0.02207 

0 

-0.0111 

-0.0196 

-0.0312 

-0.0398 

-0.0388 

-0.0428 

-0.0454 

PHM1  RGR3 

0.0332 

3,01325 

0 

-0.0085 

-0.02 

-0.0286 

-0.0277 

-0.0317 

-0.0343 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.03107 

0.01112 

0 

-0.0106 

-0.0222 

-0.0308 

-0.0298 

-0.0338 

-0.0364 

0.03532 

0.01537 

0 

-0.0063 

-0.0179 

-0.0265 

-0.0256 

-0.0295 

-0.0322 

PHM2.RGR1 

0.04165 

0.0217 

0.00846 

0 

-0.0116 

-0.0202 

-0.0192 

-0.0232 

-0.0258 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.03953 

0.01958 

0.00634 

0 

-0.0137 

-0.0223 

-0.0213 

-0.0253 

-0.028 

0.04378 

0.02383 

0.01058 

0 

-0.0095 

-0.0181 

-0.0171 

-0.0211 

-0.0237 

PHM2..RGR2 

0.05324 

0.03329 

0.02005 

0.01159 

0 

-0.0086 

-0.0076 

-0.0116 

-0.0142 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.05112 

0.03117 

0.01793 

0.00947 

0 

-0.0107 

-0.0098 

-0.0137 

-0.0164 

0.05537 

0.03542 

0.02217 

0.01371 

0 

-0.0065 

-0.0055 

-0.0095 

-0.0121 

PHM2..RGR3 

0  06183 

0.04188 

0.02864 

0.02018 

0  00859 

0 

0.00096 

-0.003 

-0.0057 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.05971 

0.03976 

0.02652 

0.01806 

0.00647 

0 

-0.0012 

-0.0051 

-0.0078 

0.06396 

0.04401 

0.03076 

0.0223 

0.01071 

0 

0.00308 

-0.0009 

-0.0035 

PH  M3  RGR1 

0.06087 

0.04092 

0.02768 

0.01922 

0.00763 

-0.001 

0 

-0.004 

-0.0066 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.05875 

0.0388 

0.02555 

0.0171 

0.00551 

-0.0031 

0 

-0.0061 

-0.0087 

0.06299 

0.04304 

0.0298 

0.02134 

0.00975 

0.00116 

0 

-0.0019 

-0.0045 

PHM3.RGR2 

0.06486 

0.04491 

0.03166 

0.02321 

0.01162 

0.00303 

0.00399 

0 

-0.0026 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.06274 

0.04279 

0.02954 

0.02108 

0.00949 

0.0009 

0.00187 

0 

-0.0048 

0.06698 

0.04703 

0.03379 

0.02533 

0.01374 

0.00515 

0.00611 

0 

-0.0005 

PHM3  RGR3 

0.06749 

0.04754 

0.03429 

0.02583 

0.01424 

0.00565 

0.00662 

0.00263 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.06537 

0.04542 

0.03217 

0.02371 

0.01212 

0.00353 

0.00449 

0.00051 

0 

0.06961 

0.04966 

0.03641 

0.02796 

0.01637 

0.00778 

0.00874 

0.00475 

0 

Level 

Least 

Sq  Mean 

PHM3.RGR3  A 
PHM3.RGR2 

B 

0.80989333 

0.80726444 

PHM2,RGR3 

C 

0.80423889 

PHM3  RGR1 

C 

0.80327667 

PHM2,RGR2 

D 

0.79  5  64889 

PHM2.RGR1 

E 

0.78405889 

PHM1,RGR3 

F 

0.77560111 

PHM1  RGR2 

G 

0.76235556 

PHM1  RGR1 

H 

0.74240444 
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Table  29.  Tukey  Test  for  Two-way  Interactions  of  PHM  Level  and  LC  Rate 


-  LSMeans  Differences  Tukey  HSD 

ot=  0.050  Q=  3.11039 


LSMeanfl 


Mean[i]-Mean(j 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

PH  MI  L 
CR1 

PHM1,L 

CR2 

PHM1L 

CR3 

PHM2.L 

CR1 

PHM2,L 

CR2 

PHM2..L 

CR3 

PHM3.L 

CR1 

PHM3..L 

CR2 

PHM3.L 

CR3 

PHM1  LCR1 

0 

-0.0461 

-0.0812 

-0.0519 

-0.0842 

-0.0948 

-0.0815 

-0.0927 

-0.0931 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

-0.0482 

-0.0833 

-0.054 

-0.0863 

-0.0969 

-0.0837 

-0.0948 

-0.0952 

0 

-0.044 

-0.0791 

-0.0498 

-0.0821 

-0.0927 

-0.0794 

-0.0906 

-0.091 

PHM1LCR2 

0.04612 

0 

-0.0351 

-0.0058 

-0.0381 

-0.0487 

-0.0354 

-0.0466 

-0.047 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.044 

0 

-0.0372 

-0.0079 

-0.0402 

-0.0508 

-0.0375 

-0.0487 

-0.0491 

0.04824 

0 

-0.0329 

-0.0036 

-0.036 

-0.0466 

-0.0333 

-0.0445 

-0.0449 

PH  Ml  LCR3 

0.08118 

0.03506 

0 

0.0293 

-0.003 

-0.0136 

-0.0004 

-0.0115 

:-ii9 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0  00068 

0.00068 

0.00068 

0.07905 

0.03293 

0 

0.02718 

-0.0052 

-0.0157 

-0.0025 

-0.0137 

-0.0141 

0.0833 

0.03718 

0 

0.03142 

-0.0009 

-0.0115 

0.00175 

-0.0094 

-0.0098 

PHM2..LCR1 

0.05187 

0.00576 

-0  0293 

0 

-0.0323 

-0.0429 

-0.0297 

-0.0408 

-0.0412 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.04975 

0.00363 

-0.0314 

0 

-0.0345 

-0.045 

-0.0318 

-0.043 

-0.0434 

0.054 

0.00788 

-0.0272 

0 

-0.0302 

-0.0408 

-0.0275 

-0.0387 

-0.0391 

PHM2.LCR2 

0.08421 

0.03809 

0.00304 

0.03234 

0 

-0.0106 

0.00267 

-0.0085 

-0.0089 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.08209 

0.03597 

0.00091 

0.03022 

0 

-0.0127 

0.00055 

-0.0106 

-0.011 

0.08633 

0.04022 

0.00516 

0.03446 

0 

-0.0085 

0.00479 

-0.0064 

-0.0068 

PHM2,LCR3 

0.09479 

0.04867 

0.01362 

0.04292 

0.01058 

0 

0.01325 

0.00208 

0.00168 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.09267 

0.04655 

0.0115 

0.0408 

0.00846 

0 

0.01113 

-4.4e-5 

-0.0004 

0.09692 

0.0508 

0.01574 

0.04504 

0.0127 

0 

0.01537 

0.0042 

0 00381 

PHM3.LCR1 

0.08154 

0.03543 

0.00037 

0.02967 

-0.0027 

-0.0132 

0 

-0.0112 

-0.0116 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.07942 

0.0333 

-0.0018 

0.02755 

-0.0048 

-0.0154 

0 

-0.0133 

-0.0137 

0.08367 

0.03755 

0.00249 

0.03179 

-0.0005 

-0.0111 

0 

-0.009 

-0.0094 

PHM3.LCR2 

0.09271 

0.0466 

0.01154 

0.04084 

0.0085 

-0.0021 

0.01117 

0 

-0.0004 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0  00068 

0.00068 

0 

0.00068 

0.09059 

0.04447 

0.00942 

0.03872 

0.00638 

-0.0042 

0.00905 

0 

-0.0025 

0.09484 

0.04872 

0.01366 

0.04296 

0.01062 

4.38e-5 

0.01329 

0 

0.00173 

PHM3.LCR3 

0.09311 

0.04699 

0.01193 

0.04123 

0.0089 

-0.0017 

0.01156 

0.00039 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.09099 

0.04487 

0.00981 

0.03911 

0.00677 

-0.0038 

0.00944 

-0.0017 

0 

0.09523 

0.04911 

0.01406 

0.04336 

0.01102 

0.00044 

0.01369 

0.00252 

0 

Level 

Least 

Sq  Mean 

PHM2.  LCR3  A 

0.81248222 

PHM3.LCR3  A 

0.81079778 

PHM3,LCR2  A 

0.81040333 

PHM2,LCR2 

B 

0.80190111 

PHM3.LCR1 

C 

0.79923333 

PHM1,LCR3 

C 

0.79886444 

PHM2.LCR1 

D 

0.76956333 

PHM1.LCR2 

E 

0.76380778 

PHM1.LCR1 

F 

0.71768889 

Levels  not  connected  by  same  letter  are  significantly  different. 
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Table  30.  Tukey  Test  for  Two-way  Interactions  of  RG  Rate  and  LC  Rate 


~  LSMeans  Differences  Tukey  HSD 

cx=  0.050  Q=  3.11039 


LSMean[j] 


Mean[i]-Mean[j‘ 
Std  Err  Dif 

Lower  CL  Dif 
Upper  CL  Dif 

RGR1..L 

CR1 

RGR1..L 

CR2 

RGR1.L 

CR3 

RGR2..L 

CR1 

RGR2..L 

CR2 

RGR2..L 

CR3 

RGR3  L 
CR1 

RGR3  L 
CR2 

RGR3..L 

CR3 

RGR1.LCR1 

0 

-0.0381 

-0.0589 

-0.0204 

-0.0484 

-0.0638 

-0.0334 

-0.0569 

-0.0668 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

-0.0403 

-0.0611 

-0.0225 

-0.0505 

-0.0659 

-0.0355 

-0.059 

-0.0689 

0 

-0.036 

-0.0568 

-0.0183 

-0.0463 

-0.0617 

-0.0313 

-0.0548 

-0.0646 

RGR1..LCR2 

0.03814 

0 

-0.0208 

0.01772 

-0.0103 

-0.0256 

0.00474 

-0.0188 

-0.0286 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.03602 

0 

-0.0229 

0.0156 

-0.0124 

-0.0278 

0.00261 

-0.0209 

-0.0308 

0.04026 

0 

-0.0187 

0.01984 

-0.0081 

-0.0235 

0.00686 

-0.0166 

-0.0265 

RGR1..LCR3 

0.05894 

0.0208 

0 

0.03852 

0.01053 

-0.0048 

0.02553 

0.00204 

-0.0078 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.05682 

0.01867 

0 

0.0364 

0.0084 

-0.007 

0.02341 

-0.0001 

-0.01 

0.06106 

0.02292 

0 

0.04064 

0.01265 

-0.0027 

0.02765 

0.00416 

-0.0057 

RGR2..LCR1 

0.02042 

-0.0177 

-0.0385 

0 

-0.028 

-0.0434 

-0.013 

-0.0365 

-0.0464 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.0183 

-0.0198 

-0.0406 

0 

-0.0301 

-0.0455 

-0.0151 

-3,3386 

-0.0485 

0.02254 

-0.0156 

-0.0364 

0 

-0.0259 

-0.0412 

-0.0109 

-0.0344 

-0.0442 

RGR2..LCR2 

0.04841 

0.01027 

-0.0105 

0.02799 

0 

-0.0154 

0.01501 

-0.0085 

-0.0184 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.04629 

0.00815 

-0.0126 

0.02587 

0 

-0.0175 

0.01288 

-0.0106 

-0.0205 

0.05053 

0.01239 

-0.0084 

0.03011 

0 

-0.0132 

0.01713 

-0.0064 

-0.0162 

RGR2..LCR3 

0.06378 

0.02563 

0.00484 

0.04336 

0.01536 

0 

0.03037 

0.00688 

-0.003 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.00068 

0.06165 

0.02351 

0.00272 

0.04123 

0.01324 

0 

0.02825 

0.00476 

-0.0051 

0.0659 

0.02776 

0.00696 

0.04548 

0.01749 

0 

0.03249 

0.009 

-0.0009 

RGR3.LCR1 

0.03341 

-0.0047 

-0.0255 

0.01299 

-0.015 

-0.0304 

0 

-0.0235 

-0.0334 

0.00068 

0.00068 

0.00068 

0  00068 

0.00068 

0.00068 

0 

0.00068 

0.00068 

0.03128 

-0.0069 

-0.0277 

0.01086 

-0.0171 

-0.0325 

0 

-0.0256 

-0.0355 

0.03553 

-0.0026 

-0.0234 

0.01511 

-0.0129 

-0.0282 

0 

-0.0214 

-0.0312 

RGR3.LCR2 

0.0569 

0.01876 

-0.002 

0.03648 

0.00849 

-0.0069 

0.02349 

0 

-0.0099 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.00068 

0.05478 

0.01663 

-0.0042 

0.03436 

0.00636 

-0.009 

0.02137 

0 

-0.012 

0.05902 

0.02088 

8.27e-5 

0.0386 

0.01061 

-0.0048 

0.02561 

0 

-0.0077 

RGR3.LCR3 

0.06677 

0.02863 

0.00783 

0.04635 

0.01836 

0.00299 

0.03336 

0.00987 

0 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0.00068 

0 

0.06465 

0.02651 

0.00571 

0.04423 

0.01624 

0.00087 

0.03124 

0.00775 

0 

0.06889 

0.03075 

0.00995 

0.04847 

0.02048 

0.00512 

0.03549 

0.01199 

0 

Least 


Level 

RGR3,LCR3  A 
RGR2..LCR3  B 
RGR1,LCR3  C 

RGR3..LCR2  C 

RGR2.LCR2  D 

RGR1,LCR2 
RGR3,LCR1 
RGR2..LCR1 
RGR1..LCR1 


Sq  Mean 

0.81099000 
0.80799667 
0.80315778 
0.80111778 
0.79263222 
0.78236222 
0.77762556 
G  0.76464000 
H  0.74422000 


Levels  not  connected  by  same  letter  are  significantly  different. 


87 


Appendix  E:  ANOVA  Results  for  MFHBFA 


Table  31.  ANOVA  of  the  MFHBFA  vs.  Maintenance  Time 


Summary  of  Fit 

RSquare 

0.108582 

RSquareAdj 

0.09797 

Root  Mean  Square  Error 

787.2074 

Mean  of  Response 

83425.51 

Observations  (or  Sum  Wgts) 

86 

Analysis  of  Variance 

Source 

DF 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

1 

6340644 

6340644 

10.2319 

Error 

84 

52054418 

619695 

Prob  >  F 

C.  Total 

85 

58395061 

0.0019* 

Table  32.  ANOVA  of  the  MFHBFA  vs.  AA  Rate 


Summary  of  Fit 

RSquare  0.050155 

RSquareAdj  0.038848 

Root  Mean  Square  Error  0.007363 

Mean  of  Response  0.495758 

Observations  (or  Sum  Wgts)  86 

Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.00024048 

0.000240 

4.4355 

Error 

84 

0.00455415 

0.000054 

Prob  >  F 

C.  Total 

85 

0.00479463 

0.0382* 

Table  33.  ANOVA  of  the  MFHBFA  vs.  FSE  Rate 


Summary  of  Fit 

RSquare  0.069577 

RSquare  Adj  0.0585 

Root  Mean  Square  Error  0.005988 

Mean  of  Response  0.69375 

Observations  (or  Sum  Wgts)  86 

Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.00022525 

0.000225 

6.2815 

Error 

84 

0.00301219 

0.000036 

Prob  >  F 

C.  Total 

85 

0.00323743 

0.0141* 
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Appendix  F:  ANOVA  Results  for  FCR 


Table  34.  ANOVA  of  the  FCR  vs.  FSE  Rate 

Summary  of  Fit 

RSquare  0.6525 

RSquare  Adj  0.648551 

Root  Mean  Square  Error  0.00642 

Mean  of  Response  0.703114 

Observations  (or  Sum  Wgts)  90 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.00680998 

0.006810 

165.2371 

Error 

88 

0.00362678 

0.000041 

Prob  >  F 

C.  Total 

89 

0.01043675 

<.0001* 

Table  35.  ANOVA  of  the  FCR  vs.  AA  Rate 


Summary  of  Fit 

RSquare 

0.67558 

RSquare  Adj 

0.671893 

Root  Mean  Square  Error 

0.007981 

Mean  of  Response 

0.507963 

Observations  (or  Sum  Wgts) 

90 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.01167293 

0.011673 

183.2529 

Error 

88 

0.00560546 

0.000064 

Prob  >  F 

C.  Total 

89 

0.01727839 

<.0001* 
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Appendix  G:  ANOVA  Results  for  CFDR 


Table  36.  ANOVA  of  the  CFDR  vs.  FSE  Rate 


Summary  of  Fit 

RSquare 

0.834445 

RSquare  Adj 

0.832564 

Root  Mean  Square  Error 

0.005443 

Mean  of  Response 

0.708038 

Observations  (or  Sum  Wgts) 

90 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.01314240 

0.013142 

443.5451 

Error 

88 

0.00260747 

0.000030 

Prob  >  F 

C.  Total 

89 

0.01574987 

<.0001* 

Table  37.  ANOVA  of  the  CFDR  vs.  AA  Rate 


Summary  of  Fit 

RSquare 

0.828092 

RSquare  Adj 

0.826139 

Root  Mean  Square  Error 

0.007798 

Mean  of  Response 

0.515607 

Observations  (or  Sum  Wgts) 

90 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.02577568 

0.025776 

423.9031 

Error 

88 

0.00535089 

0.000061 

Prob  >  F 

C.  Total 

89 

0.03112658 

<.0001* 

90 


Appendix  H:  ANOVA  Results  for  FIR 

Table  38.  ANOVA  of  the  FIR  vs.  FSE  Rate 


Summary  of  Fit 

RSquare 

0.776166 

RSquare  Adj 

0.773623 

Root  Mean  Square  Error 

0.006142 

Mean  of  Response 

0.707297 

Observations  (or  Sum  Wgts) 

90 

Analysis  of  Variance 

Source 

DF 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.01151292 

0.011513 

305.1493 

Error 

88 

0.00332013 

0.000038 

Prob  >  F 

C.  Total 

89 

0.01483305 

<.0001* 

Table  39.  ANOVA  of  the  FIR  vs.  AA  Rate 


Summary  of  Fit 

RSquare 

0.766916 

RSquare  Adj 

0.764268 

Root  Mean  Square  Error 

0.008871 

Mean  of  Response 

0.51432 2 

Observations  (or  Sum  Wgts) 

90 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.02278392 

0.022784 

289.5470 

Error 

88 

0.00692456 

0.000079 

Prob  >  F 

C.  Total 

89 

0.02970848 

<.ooor 
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Appendix  I:  ANOVA  Results  for  KFAR 


Table  40.  ANOVA  of  the  KFAR  vs.  FSE  Rate 


Summary  of  Fit 

RSquare 

0.962543 

RSquare  Adj 

0.962117 

Root  Mean  Square  Error 

0.005628 

Mean  of  Response 

0.728201 

Observations  (or  Sum  Wgts) 

90 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.07161524 

0.071615 

2261.369 

Error 

88 

0.00278687 

0.000032 

Prob  >  F 

C.  Total 

89 

0.07440211 

<.0001* 

Table  41.  ANOVA  of  the  KFAR  vs.  AA  Rate 


Summary  of  Fit 

RSquare 

0.938323 

RSquare  Adj 

0.937622 

Root  Mean  Square  Error 

0.009816 

Mean  of  Response 

0.545301 

Observations  (or  Sum  Wgts) 

90 

Analysis  of  Variance 

Source 

DF 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

1 

0.12899207 

0.128992 

1338.795 

Error 

88 

0.00847874 

0.000096 

Prob  >  F 

C.  Total 

89 

0.13747081 

<.0001* 
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